HADOOP-4687 More moving around
git-svn-id: https://svn.apache.org/repos/asf/hadoop/core/branches/HADOOP-4687/core@776175 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Owen O'Malley
parent
5128a9a453
commit
c94ff0f240
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,244 @@
|
||||||
|
|
||||||
|
Apache License
|
||||||
|
Version 2.0, January 2004
|
||||||
|
http://www.apache.org/licenses/
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
|
|
||||||
|
1. Definitions.
|
||||||
|
|
||||||
|
"License" shall mean the terms and conditions for use, reproduction,
|
||||||
|
and distribution as defined by Sections 1 through 9 of this document.
|
||||||
|
|
||||||
|
"Licensor" shall mean the copyright owner or entity authorized by
|
||||||
|
the copyright owner that is granting the License.
|
||||||
|
|
||||||
|
"Legal Entity" shall mean the union of the acting entity and all
|
||||||
|
other entities that control, are controlled by, or are under common
|
||||||
|
control with that entity. For the purposes of this definition,
|
||||||
|
"control" means (i) the power, direct or indirect, to cause the
|
||||||
|
direction or management of such entity, whether by contract or
|
||||||
|
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||||
|
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||||
|
|
||||||
|
"You" (or "Your") shall mean an individual or Legal Entity
|
||||||
|
exercising permissions granted by this License.
|
||||||
|
|
||||||
|
"Source" form shall mean the preferred form for making modifications,
|
||||||
|
including but not limited to software source code, documentation
|
||||||
|
source, and configuration files.
|
||||||
|
|
||||||
|
"Object" form shall mean any form resulting from mechanical
|
||||||
|
transformation or translation of a Source form, including but
|
||||||
|
not limited to compiled object code, generated documentation,
|
||||||
|
and conversions to other media types.
|
||||||
|
|
||||||
|
"Work" shall mean the work of authorship, whether in Source or
|
||||||
|
Object form, made available under the License, as indicated by a
|
||||||
|
copyright notice that is included in or attached to the work
|
||||||
|
(an example is provided in the Appendix below).
|
||||||
|
|
||||||
|
"Derivative Works" shall mean any work, whether in Source or Object
|
||||||
|
form, that is based on (or derived from) the Work and for which the
|
||||||
|
editorial revisions, annotations, elaborations, or other modifications
|
||||||
|
represent, as a whole, an original work of authorship. For the purposes
|
||||||
|
of this License, Derivative Works shall not include works that remain
|
||||||
|
separable from, or merely link (or bind by name) to the interfaces of,
|
||||||
|
the Work and Derivative Works thereof.
|
||||||
|
|
||||||
|
"Contribution" shall mean any work of authorship, including
|
||||||
|
the original version of the Work and any modifications or additions
|
||||||
|
to that Work or Derivative Works thereof, that is intentionally
|
||||||
|
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||||
|
or by an individual or Legal Entity authorized to submit on behalf of
|
||||||
|
the copyright owner. For the purposes of this definition, "submitted"
|
||||||
|
means any form of electronic, verbal, or written communication sent
|
||||||
|
to the Licensor or its representatives, including but not limited to
|
||||||
|
communication on electronic mailing lists, source code control systems,
|
||||||
|
and issue tracking systems that are managed by, or on behalf of, the
|
||||||
|
Licensor for the purpose of discussing and improving the Work, but
|
||||||
|
excluding communication that is conspicuously marked or otherwise
|
||||||
|
designated in writing by the copyright owner as "Not a Contribution."
|
||||||
|
|
||||||
|
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||||
|
on behalf of whom a Contribution has been received by Licensor and
|
||||||
|
subsequently incorporated within the Work.
|
||||||
|
|
||||||
|
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
copyright license to reproduce, prepare Derivative Works of,
|
||||||
|
publicly display, publicly perform, sublicense, and distribute the
|
||||||
|
Work and such Derivative Works in Source or Object form.
|
||||||
|
|
||||||
|
3. Grant of Patent License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
(except as stated in this section) patent license to make, have made,
|
||||||
|
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||||
|
where such license applies only to those patent claims licensable
|
||||||
|
by such Contributor that are necessarily infringed by their
|
||||||
|
Contribution(s) alone or by combination of their Contribution(s)
|
||||||
|
with the Work to which such Contribution(s) was submitted. If You
|
||||||
|
institute patent litigation against any entity (including a
|
||||||
|
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||||
|
or a Contribution incorporated within the Work constitutes direct
|
||||||
|
or contributory patent infringement, then any patent licenses
|
||||||
|
granted to You under this License for that Work shall terminate
|
||||||
|
as of the date such litigation is filed.
|
||||||
|
|
||||||
|
4. Redistribution. You may reproduce and distribute copies of the
|
||||||
|
Work or Derivative Works thereof in any medium, with or without
|
||||||
|
modifications, and in Source or Object form, provided that You
|
||||||
|
meet the following conditions:
|
||||||
|
|
||||||
|
(a) You must give any other recipients of the Work or
|
||||||
|
Derivative Works a copy of this License; and
|
||||||
|
|
||||||
|
(b) You must cause any modified files to carry prominent notices
|
||||||
|
stating that You changed the files; and
|
||||||
|
|
||||||
|
(c) You must retain, in the Source form of any Derivative Works
|
||||||
|
that You distribute, all copyright, patent, trademark, and
|
||||||
|
attribution notices from the Source form of the Work,
|
||||||
|
excluding those notices that do not pertain to any part of
|
||||||
|
the Derivative Works; and
|
||||||
|
|
||||||
|
(d) If the Work includes a "NOTICE" text file as part of its
|
||||||
|
distribution, then any Derivative Works that You distribute must
|
||||||
|
include a readable copy of the attribution notices contained
|
||||||
|
within such NOTICE file, excluding those notices that do not
|
||||||
|
pertain to any part of the Derivative Works, in at least one
|
||||||
|
of the following places: within a NOTICE text file distributed
|
||||||
|
as part of the Derivative Works; within the Source form or
|
||||||
|
documentation, if provided along with the Derivative Works; or,
|
||||||
|
within a display generated by the Derivative Works, if and
|
||||||
|
wherever such third-party notices normally appear. The contents
|
||||||
|
of the NOTICE file are for informational purposes only and
|
||||||
|
do not modify the License. You may add Your own attribution
|
||||||
|
notices within Derivative Works that You distribute, alongside
|
||||||
|
or as an addendum to the NOTICE text from the Work, provided
|
||||||
|
that such additional attribution notices cannot be construed
|
||||||
|
as modifying the License.
|
||||||
|
|
||||||
|
You may add Your own copyright statement to Your modifications and
|
||||||
|
may provide additional or different license terms and conditions
|
||||||
|
for use, reproduction, or distribution of Your modifications, or
|
||||||
|
for any such Derivative Works as a whole, provided Your use,
|
||||||
|
reproduction, and distribution of the Work otherwise complies with
|
||||||
|
the conditions stated in this License.
|
||||||
|
|
||||||
|
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||||
|
any Contribution intentionally submitted for inclusion in the Work
|
||||||
|
by You to the Licensor shall be under the terms and conditions of
|
||||||
|
this License, without any additional terms or conditions.
|
||||||
|
Notwithstanding the above, nothing herein shall supersede or modify
|
||||||
|
the terms of any separate license agreement you may have executed
|
||||||
|
with Licensor regarding such Contributions.
|
||||||
|
|
||||||
|
6. Trademarks. This License does not grant permission to use the trade
|
||||||
|
names, trademarks, service marks, or product names of the Licensor,
|
||||||
|
except as required for reasonable and customary use in describing the
|
||||||
|
origin of the Work and reproducing the content of the NOTICE file.
|
||||||
|
|
||||||
|
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||||
|
agreed to in writing, Licensor provides the Work (and each
|
||||||
|
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
implied, including, without limitation, any warranties or conditions
|
||||||
|
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||||
|
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||||
|
appropriateness of using or redistributing the Work and assume any
|
||||||
|
risks associated with Your exercise of permissions under this License.
|
||||||
|
|
||||||
|
8. Limitation of Liability. In no event and under no legal theory,
|
||||||
|
whether in tort (including negligence), contract, or otherwise,
|
||||||
|
unless required by applicable law (such as deliberate and grossly
|
||||||
|
negligent acts) or agreed to in writing, shall any Contributor be
|
||||||
|
liable to You for damages, including any direct, indirect, special,
|
||||||
|
incidental, or consequential damages of any character arising as a
|
||||||
|
result of this License or out of the use or inability to use the
|
||||||
|
Work (including but not limited to damages for loss of goodwill,
|
||||||
|
work stoppage, computer failure or malfunction, or any and all
|
||||||
|
other commercial damages or losses), even if such Contributor
|
||||||
|
has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
9. Accepting Warranty or Additional Liability. While redistributing
|
||||||
|
the Work or Derivative Works thereof, You may choose to offer,
|
||||||
|
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||||
|
or other liability obligations and/or rights consistent with this
|
||||||
|
License. However, in accepting such obligations, You may act only
|
||||||
|
on Your own behalf and on Your sole responsibility, not on behalf
|
||||||
|
of any other Contributor, and only if You agree to indemnify,
|
||||||
|
defend, and hold each Contributor harmless for any liability
|
||||||
|
incurred by, or claims asserted against, such Contributor by reason
|
||||||
|
of your accepting any such warranty or additional liability.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
APPENDIX: How to apply the Apache License to your work.
|
||||||
|
|
||||||
|
To apply the Apache License to your work, attach the following
|
||||||
|
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||||
|
replaced with your own identifying information. (Don't include
|
||||||
|
the brackets!) The text should be enclosed in the appropriate
|
||||||
|
comment syntax for the file format. We also recommend that a
|
||||||
|
file or class name and description of purpose be included on the
|
||||||
|
same "printed page" as the copyright notice for easier
|
||||||
|
identification within third-party archives.
|
||||||
|
|
||||||
|
Copyright [yyyy] [name of copyright owner]
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
APACHE HADOOP SUBCOMPONENTS:
|
||||||
|
|
||||||
|
The Apache Hadoop project contains subcomponents with separate copyright
|
||||||
|
notices and license terms. Your use of the source code for the these
|
||||||
|
subcomponents is subject to the terms and conditions of the following
|
||||||
|
licenses.
|
||||||
|
|
||||||
|
For the org.apache.hadoop.util.bloom.* classes:
|
||||||
|
|
||||||
|
/**
|
||||||
|
*
|
||||||
|
* Copyright (c) 2005, European Commission project OneLab under contract
|
||||||
|
* 034819 (http://www.one-lab.org)
|
||||||
|
* All rights reserved.
|
||||||
|
* Redistribution and use in source and binary forms, with or
|
||||||
|
* without modification, are permitted provided that the following
|
||||||
|
* conditions are met:
|
||||||
|
* - Redistributions of source code must retain the above copyright
|
||||||
|
* notice, this list of conditions and the following disclaimer.
|
||||||
|
* - Redistributions in binary form must reproduce the above copyright
|
||||||
|
* notice, this list of conditions and the following disclaimer in
|
||||||
|
* the documentation and/or other materials provided with the distribution.
|
||||||
|
* - Neither the name of the University Catholique de Louvain - UCL
|
||||||
|
* nor the names of its contributors may be used to endorse or
|
||||||
|
* promote products derived from this software without specific prior
|
||||||
|
* written permission.
|
||||||
|
*
|
||||||
|
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||||
|
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||||
|
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
|
||||||
|
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
|
||||||
|
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
|
||||||
|
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
|
||||||
|
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
|
||||||
|
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||||
|
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
||||||
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
|
||||||
|
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
||||||
|
* POSSIBILITY OF SUCH DAMAGE.
|
||||||
|
*/
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
This product includes software developed by The Apache Software
|
||||||
|
Foundation (http://www.apache.org/).
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
For the latest information about Hadoop, please visit our website at:
|
||||||
|
|
||||||
|
http://hadoop.apache.org/core/
|
||||||
|
|
||||||
|
and our wiki, at:
|
||||||
|
|
||||||
|
http://wiki.apache.org/hadoop/
|
||||||
|
|
||||||
|
This distribution includes cryptographic software. The country in
|
||||||
|
which you currently reside may have restrictions on the import,
|
||||||
|
possession, use, and/or re-export to another country, of
|
||||||
|
encryption software. BEFORE using any encryption software, please
|
||||||
|
check your country's laws, regulations and policies concerning the
|
||||||
|
import, possession, or use, and re-export of encryption software, to
|
||||||
|
see if this is permitted. See <http://www.wassenaar.org/> for more
|
||||||
|
information.
|
||||||
|
|
||||||
|
The U.S. Government Department of Commerce, Bureau of Industry and
|
||||||
|
Security (BIS), has classified this software as Export Commodity
|
||||||
|
Control Number (ECCN) 5D002.C.1, which includes information security
|
||||||
|
software using or performing cryptographic functions with asymmetric
|
||||||
|
algorithms. The form and manner of this Apache Software Foundation
|
||||||
|
distribution makes it eligible for export under the License Exception
|
||||||
|
ENC Technology Software Unrestricted (TSU) exception (see the BIS
|
||||||
|
Export Administration Regulations, Section 740.13) for both object
|
||||||
|
code and source code.
|
||||||
|
|
||||||
|
The following provides more details on the included cryptographic
|
||||||
|
software:
|
||||||
|
Hadoop Core uses the SSL libraries from the Jetty project written
|
||||||
|
by mortbay.org.
|
||||||
|
|
@ -0,0 +1,108 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
# This script runs the hadoop core commands.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hadoop-config.sh
|
||||||
|
|
||||||
|
function print_usage(){
|
||||||
|
echo "Usage: hadoop [--config confdir] COMMAND"
|
||||||
|
echo " where COMMAND is one of:"
|
||||||
|
echo " fs run a generic filesystem user client"
|
||||||
|
echo " version print the version"
|
||||||
|
echo " jar <jar> run a jar file"
|
||||||
|
echo " distcp <srcurl> <desturl> copy file or directories recursively"
|
||||||
|
echo " archive -archiveName NAME <src>* <dest> create a hadoop archive"
|
||||||
|
echo " daemonlog get/set the log level for each daemon"
|
||||||
|
echo " or"
|
||||||
|
echo " CLASSNAME run the class named CLASSNAME"
|
||||||
|
echo ""
|
||||||
|
echo "Most commands print help when invoked w/o parameters."
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ $# = 0 ]; then
|
||||||
|
print_usage
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
|
||||||
|
COMMAND=$1
|
||||||
|
case $COMMAND in
|
||||||
|
#hdfs commands
|
||||||
|
namenode|secondarynamenode|datanode|dfs|dfsadmin|fsck|balancer)
|
||||||
|
echo "DEPRECATED: Use of this script to execute hdfs command is deprecated."
|
||||||
|
echo "Instead use the hdfs command for it."
|
||||||
|
echo ""
|
||||||
|
#try to locate hdfs and if present, delegate to it.
|
||||||
|
if [ -f "${HADOOP_HDFS_HOME}"/bin/hdfs ]; then
|
||||||
|
exec "${HADOOP_HDFS_HOME}"/bin/hdfs $*
|
||||||
|
else
|
||||||
|
echo "HDFS not found."
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
|
||||||
|
#mapred commands
|
||||||
|
mradmin|jobtracker|tasktracker|pipes|job|queue)
|
||||||
|
echo "DEPRECATED: Use of this script to execute mapred command is deprecated."
|
||||||
|
echo "Instead use the mapred command for it."
|
||||||
|
echo ""
|
||||||
|
#try to locate mapred and if present, delegate to it.
|
||||||
|
if [ -f "${HADOOP_MAPRED_HOME}"/bin/mapred ]; then
|
||||||
|
exec "${HADOOP_MAPRED_HOME}"/bin/mapred $*
|
||||||
|
else
|
||||||
|
echo "MAPRED not found."
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
|
||||||
|
#core commands
|
||||||
|
*)
|
||||||
|
# the core commands
|
||||||
|
if [ "$COMMAND" = "fs" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.fs.FsShell
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "version" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.util.VersionInfo
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "jar" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.util.RunJar
|
||||||
|
elif [ "$COMMAND" = "distcp" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.tools.DistCp
|
||||||
|
CLASSPATH=${CLASSPATH}:${TOOL_PATH}
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "daemonlog" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.log.LogLevel
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "archive" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.tools.HadoopArchives
|
||||||
|
CLASSPATH=${CLASSPATH}:${TOOL_PATH}
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
else
|
||||||
|
CLASS=$COMMAND
|
||||||
|
fi
|
||||||
|
shift
|
||||||
|
|
||||||
|
if $cygwin; then
|
||||||
|
CLASSPATH=`cygpath -p -w "$CLASSPATH"`
|
||||||
|
fi
|
||||||
|
exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
|
||||||
|
;;
|
||||||
|
|
||||||
|
esac
|
||||||
|
|
@ -0,0 +1,248 @@
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
# included in all the hadoop scripts with source command
|
||||||
|
# should not be executable directly
|
||||||
|
# also should not be passed any arguments, since we need original $*
|
||||||
|
|
||||||
|
# resolve links - $0 may be a softlink
|
||||||
|
|
||||||
|
this="$0"
|
||||||
|
while [ -h "$this" ]; do
|
||||||
|
ls=`ls -ld "$this"`
|
||||||
|
link=`expr "$ls" : '.*-> \(.*\)$'`
|
||||||
|
if expr "$link" : '.*/.*' > /dev/null; then
|
||||||
|
this="$link"
|
||||||
|
else
|
||||||
|
this=`dirname "$this"`/"$link"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# convert relative path to absolute path
|
||||||
|
bin=`dirname "$this"`
|
||||||
|
script=`basename "$this"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
this="$bin/$script"
|
||||||
|
|
||||||
|
# the root of the Hadoop installation
|
||||||
|
#TODO: change the env variable when dir structure is changed
|
||||||
|
export HADOOP_HOME=`dirname "$this"`/..
|
||||||
|
export HADOOP_CORE_HOME="${HADOOP_HOME}"
|
||||||
|
#export HADOOP_HOME=`dirname "$this"`/../..
|
||||||
|
#export HADOOP_CORE_HOME="${HADOOP_CORE_HOME:-`dirname "$this"`/..}"
|
||||||
|
|
||||||
|
#check to see if the conf dir is given as an optional argument
|
||||||
|
if [ $# -gt 1 ]
|
||||||
|
then
|
||||||
|
if [ "--config" = "$1" ]
|
||||||
|
then
|
||||||
|
shift
|
||||||
|
confdir=$1
|
||||||
|
shift
|
||||||
|
HADOOP_CONF_DIR=$confdir
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Allow alternate conf dir location.
|
||||||
|
export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/conf}"
|
||||||
|
|
||||||
|
#check to see it is specified whether to use the slaves or the
|
||||||
|
# masters file
|
||||||
|
if [ $# -gt 1 ]
|
||||||
|
then
|
||||||
|
if [ "--hosts" = "$1" ]
|
||||||
|
then
|
||||||
|
shift
|
||||||
|
slavesfile=$1
|
||||||
|
shift
|
||||||
|
export HADOOP_SLAVES="${HADOOP_CONF_DIR}/$slavesfile"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
cygwin=false
|
||||||
|
case "`uname`" in
|
||||||
|
CYGWIN*) cygwin=true;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then
|
||||||
|
. "${HADOOP_CONF_DIR}/hadoop-env.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# some Java parameters
|
||||||
|
if [ "$JAVA_HOME" != "" ]; then
|
||||||
|
#echo "run java in $JAVA_HOME"
|
||||||
|
JAVA_HOME=$JAVA_HOME
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$JAVA_HOME" = "" ]; then
|
||||||
|
echo "Error: JAVA_HOME is not set."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
JAVA=$JAVA_HOME/bin/java
|
||||||
|
JAVA_HEAP_MAX=-Xmx1000m
|
||||||
|
|
||||||
|
# check envvars which might override default args
|
||||||
|
if [ "$HADOOP_HEAPSIZE" != "" ]; then
|
||||||
|
#echo "run with heapsize $HADOOP_HEAPSIZE"
|
||||||
|
JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
|
||||||
|
#echo $JAVA_HEAP_MAX
|
||||||
|
fi
|
||||||
|
|
||||||
|
# CLASSPATH initially contains $HADOOP_CONF_DIR
|
||||||
|
CLASSPATH="${HADOOP_CONF_DIR}"
|
||||||
|
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
|
||||||
|
|
||||||
|
# for developers, add Hadoop classes to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_CORE_HOME/build/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME/build/classes
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_CORE_HOME/build/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME/build
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_CORE_HOME/build/test/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME/build/test/classes
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_CORE_HOME/build/tools" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME/build/tools
|
||||||
|
fi
|
||||||
|
|
||||||
|
# so that filenames w/ spaces are handled correctly in loops below
|
||||||
|
IFS=
|
||||||
|
|
||||||
|
# for releases, add core hadoop jar & webapps to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_CORE_HOME/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME
|
||||||
|
fi
|
||||||
|
for f in $HADOOP_CORE_HOME/hadoop-*-core.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
# add libs to CLASSPATH
|
||||||
|
for f in $HADOOP_CORE_HOME/lib/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -d "$HADOOP_CORE_HOME/build/ivy/lib/Hadoop/common" ]; then
|
||||||
|
for f in $HADOOP_CORE_HOME/build/ivy/lib/Hadoop/common/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
for f in $HADOOP_CORE_HOME/lib/jsp-2.1/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
for f in $HADOOP_CORE_HOME/hadoop-*-tools.jar; do
|
||||||
|
TOOL_PATH=${TOOL_PATH}:$f;
|
||||||
|
done
|
||||||
|
for f in $HADOOP_CORE_HOME/build/hadoop-*-tools.jar; do
|
||||||
|
TOOL_PATH=${TOOL_PATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
# add user-specified CLASSPATH last
|
||||||
|
if [ "$HADOOP_CLASSPATH" != "" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:${HADOOP_CLASSPATH}
|
||||||
|
fi
|
||||||
|
|
||||||
|
# default log directory & file
|
||||||
|
if [ "$HADOOP_LOG_DIR" = "" ]; then
|
||||||
|
HADOOP_LOG_DIR="$HADOOP_HOME/logs"
|
||||||
|
fi
|
||||||
|
if [ "$HADOOP_LOGFILE" = "" ]; then
|
||||||
|
HADOOP_LOGFILE='hadoop.log'
|
||||||
|
fi
|
||||||
|
|
||||||
|
# default policy file for service-level authorization
|
||||||
|
if [ "$HADOOP_POLICYFILE" = "" ]; then
|
||||||
|
HADOOP_POLICYFILE="hadoop-policy.xml"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# restore ordinary behaviour
|
||||||
|
unset IFS
|
||||||
|
|
||||||
|
# cygwin path translation
|
||||||
|
if $cygwin; then
|
||||||
|
HADOOP_CORE_HOME=`cygpath -w "$HADOOP_CORE_HOME"`
|
||||||
|
HADOOP_LOG_DIR=`cygpath -w "$HADOOP_LOG_DIR"`
|
||||||
|
TOOL_PATH=`cygpath -p -w "$TOOL_PATH"`
|
||||||
|
fi
|
||||||
|
# setup 'java.library.path' for native-hadoop code if necessary
|
||||||
|
JAVA_LIBRARY_PATH=''
|
||||||
|
if [ -d "${HADOOP_CORE_HOME}/build/native" -o -d "${HADOOP_CORE_HOME}/lib/native" ]; then
|
||||||
|
JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
|
||||||
|
|
||||||
|
if [ -d "$HADOOP_CORE_HOME/build/native" ]; then
|
||||||
|
JAVA_LIBRARY_PATH=${HADOOP_CORE_HOME}/build/native/${JAVA_PLATFORM}/lib
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -d "${HADOOP_CORE_HOME}/lib/native" ]; then
|
||||||
|
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
|
||||||
|
JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_CORE_HOME}/lib/native/${JAVA_PLATFORM}
|
||||||
|
else
|
||||||
|
JAVA_LIBRARY_PATH=${HADOOP_CORE_HOME}/lib/native/${JAVA_PLATFORM}
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# cygwin path translation
|
||||||
|
if $cygwin; then
|
||||||
|
JAVA_LIBRARY_PATH=`cygpath -p "$JAVA_LIBRARY_PATH"`
|
||||||
|
fi
|
||||||
|
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.dir=$HADOOP_LOG_DIR"
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.file=$HADOOP_LOGFILE"
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.home.dir=$HADOOP_CORE_HOME"
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.id.str=$HADOOP_IDENT_STRING"
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.root.logger=${HADOOP_ROOT_LOGGER:-INFO,console}"
|
||||||
|
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
|
||||||
|
fi
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.policy.file=$HADOOP_POLICYFILE"
|
||||||
|
|
||||||
|
# put hdfs in classpath if present
|
||||||
|
if [ "$HADOOP_HDFS_HOME" = "" ]; then
|
||||||
|
if [ -d "${HADOOP_HOME}/hdfs" ]; then
|
||||||
|
HADOOP_HDFS_HOME=$HADOOP_HOME/hdfs
|
||||||
|
echo Found HDFS installed at $HADOOP_HDFS_HOME
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -d "${HADOOP_HDFS_HOME}" ]; then
|
||||||
|
for f in $HADOOP_HDFS_HOME/hadoop-*-hdfs.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
# add libs to CLASSPATH
|
||||||
|
for f in $HADOOP_HDFS_HOME/lib/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -d "$HADOOP_HDFS_HOME/build/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HDFS_HOME/build/classes
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# set mapred home if mapred is present
|
||||||
|
if [ "$HADOOP_MAPRED_HOME" = "" ]; then
|
||||||
|
if [ -d "${HADOOP_HOME}/mapred" ]; then
|
||||||
|
HADOOP_MAPRED_HOME=$HADOOP_HOME/mapred
|
||||||
|
echo Found MAPRED installed at $HADOOP_MAPRED_HOME
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# TODO:remove this when dir structure is changed
|
||||||
|
export HADOOP_HDFS_HOME=$HADOOP_HOME
|
||||||
|
export HADOOP_MAPRED_HOME=$HADOOP_HOME
|
||||||
|
|
@ -0,0 +1,152 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Runs a Hadoop command as a daemon.
|
||||||
|
#
|
||||||
|
# Environment Variables
|
||||||
|
#
|
||||||
|
# HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_HOME}/conf.
|
||||||
|
# HADOOP_LOG_DIR Where log files are stored. PWD by default.
|
||||||
|
# HADOOP_MASTER host:path where hadoop code should be rsync'd from
|
||||||
|
# HADOOP_PID_DIR The pid files are stored. /tmp by default.
|
||||||
|
# HADOOP_IDENT_STRING A string representing this instance of hadoop. $USER by default
|
||||||
|
# HADOOP_NICENESS The scheduling priority for daemons. Defaults to 0.
|
||||||
|
##
|
||||||
|
|
||||||
|
usage="Usage: hadoop-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] [--script script] (start|stop) <hadoop-command> <args...>"
|
||||||
|
|
||||||
|
# if no args specified, show usage
|
||||||
|
if [ $# -le 1 ]; then
|
||||||
|
echo $usage
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hadoop-config.sh
|
||||||
|
|
||||||
|
# get arguments
|
||||||
|
|
||||||
|
#default value
|
||||||
|
hadoopScript="$HADOOP_HOME"/bin/hadoop
|
||||||
|
if [ "--script" = "$1" ]
|
||||||
|
then
|
||||||
|
shift
|
||||||
|
hadoopScript=$1
|
||||||
|
shift
|
||||||
|
fi
|
||||||
|
startStop=$1
|
||||||
|
shift
|
||||||
|
command=$1
|
||||||
|
shift
|
||||||
|
|
||||||
|
hadoop_rotate_log ()
|
||||||
|
{
|
||||||
|
log=$1;
|
||||||
|
num=5;
|
||||||
|
if [ -n "$2" ]; then
|
||||||
|
num=$2
|
||||||
|
fi
|
||||||
|
if [ -f "$log" ]; then # rotate logs
|
||||||
|
while [ $num -gt 1 ]; do
|
||||||
|
prev=`expr $num - 1`
|
||||||
|
[ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num"
|
||||||
|
num=$prev
|
||||||
|
done
|
||||||
|
mv "$log" "$log.$num";
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then
|
||||||
|
. "${HADOOP_CONF_DIR}/hadoop-env.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# get log directory
|
||||||
|
if [ "$HADOOP_LOG_DIR" = "" ]; then
|
||||||
|
export HADOOP_LOG_DIR="$HADOOP_HOME/logs"
|
||||||
|
fi
|
||||||
|
mkdir -p "$HADOOP_LOG_DIR"
|
||||||
|
|
||||||
|
if [ "$HADOOP_PID_DIR" = "" ]; then
|
||||||
|
HADOOP_PID_DIR=/tmp
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$HADOOP_IDENT_STRING" = "" ]; then
|
||||||
|
export HADOOP_IDENT_STRING="$USER"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# some variables
|
||||||
|
export HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-$HOSTNAME.log
|
||||||
|
export HADOOP_ROOT_LOGGER="INFO,DRFA"
|
||||||
|
log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-$HOSTNAME.out
|
||||||
|
pid=$HADOOP_PID_DIR/hadoop-$HADOOP_IDENT_STRING-$command.pid
|
||||||
|
|
||||||
|
# Set default scheduling priority
|
||||||
|
if [ "$HADOOP_NICENESS" = "" ]; then
|
||||||
|
export HADOOP_NICENESS=0
|
||||||
|
fi
|
||||||
|
|
||||||
|
case $startStop in
|
||||||
|
|
||||||
|
(start)
|
||||||
|
|
||||||
|
mkdir -p "$HADOOP_PID_DIR"
|
||||||
|
|
||||||
|
if [ -f $pid ]; then
|
||||||
|
if kill -0 `cat $pid` > /dev/null 2>&1; then
|
||||||
|
echo $command running as process `cat $pid`. Stop it first.
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$HADOOP_MASTER" != "" ]; then
|
||||||
|
echo rsync from $HADOOP_MASTER
|
||||||
|
rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' $HADOOP_MASTER/ "$HADOOP_HOME"
|
||||||
|
fi
|
||||||
|
|
||||||
|
hadoop_rotate_log $log
|
||||||
|
echo starting $command, logging to $log
|
||||||
|
cd "$HADOOP_HOME"
|
||||||
|
nohup nice -n $HADOOP_NICENESS $hadoopScript --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
|
||||||
|
echo $! > $pid
|
||||||
|
sleep 1; head "$log"
|
||||||
|
;;
|
||||||
|
|
||||||
|
(stop)
|
||||||
|
|
||||||
|
if [ -f $pid ]; then
|
||||||
|
if kill -0 `cat $pid` > /dev/null 2>&1; then
|
||||||
|
echo stopping $command
|
||||||
|
kill `cat $pid`
|
||||||
|
else
|
||||||
|
echo no $command to stop
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo no $command to stop
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
|
||||||
|
(*)
|
||||||
|
echo $usage
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
|
||||||
|
esac
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Run a Hadoop command on all slave hosts.
|
||||||
|
|
||||||
|
usage="Usage: hadoop-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command args..."
|
||||||
|
|
||||||
|
# if no args specified, show usage
|
||||||
|
if [ $# -le 1 ]; then
|
||||||
|
echo $usage
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. $bin/hadoop-config.sh
|
||||||
|
|
||||||
|
exec "$bin/slaves.sh" --config $HADOOP_CONF_DIR cd "$HADOOP_HOME" \; "$bin/hadoop-daemon.sh" --config $HADOOP_CONF_DIR "$@"
|
||||||
|
|
@ -0,0 +1,110 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hdfs-config.sh
|
||||||
|
|
||||||
|
function print_usage(){
|
||||||
|
echo "Usage: hdfs [--config confdir] COMMAND"
|
||||||
|
echo " where COMMAND is one of:"
|
||||||
|
echo " namenode -format format the DFS filesystem"
|
||||||
|
echo " secondarynamenode run the DFS secondary namenode"
|
||||||
|
echo " namenode run the DFS namenode"
|
||||||
|
echo " datanode run a DFS datanode"
|
||||||
|
echo " dfsadmin run a DFS admin client"
|
||||||
|
echo " fsck run a DFS filesystem checking utility"
|
||||||
|
echo " balancer run a cluster balancing utility"
|
||||||
|
echo " jmxget get JMX exported values from NameNode or DataNode."
|
||||||
|
echo " oiv apply the offline fsimage viewer to an fsimage"
|
||||||
|
echo " Use -help to see options"
|
||||||
|
echo ""
|
||||||
|
echo "Most commands print help when invoked w/o parameters."
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ $# = 0 ]; then
|
||||||
|
print_usage
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
|
||||||
|
COMMAND=$1
|
||||||
|
shift
|
||||||
|
|
||||||
|
if [ "$COMMAND" = "namenode" ] ; then
|
||||||
|
CLASS='org.apache.hadoop.hdfs.server.namenode.NameNode'
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NAMENODE_OPTS"
|
||||||
|
elif [ "$COMMAND" = "secondarynamenode" ] ; then
|
||||||
|
CLASS='org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode'
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_SECONDARYNAMENODE_OPTS"
|
||||||
|
elif [ "$COMMAND" = "datanode" ] ; then
|
||||||
|
CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode'
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_DATANODE_OPTS"
|
||||||
|
elif [ "$COMMAND" = "dfs" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.fs.FsShell
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "dfsadmin" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.hdfs.tools.DFSAdmin
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "fsck" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.hdfs.tools.DFSck
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "balancer" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.hdfs.server.balancer.Balancer
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_BALANCER_OPTS"
|
||||||
|
elif [ "$COMMAND" = "jmxget" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.hdfs.tools.JMXGet
|
||||||
|
elif [ "$COMMAND" = "oiv" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer
|
||||||
|
else
|
||||||
|
echo $COMMAND - invalid command
|
||||||
|
print_usage
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
|
||||||
|
# for developers, add hdfs classes to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_HDFS_HOME/build/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HDFS_HOME/build/classes
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_HDFS_HOME/build/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HDFS_HOME/build
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_HDFS_HOME/build/test/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HDFS_HOME/build/test/classes
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_HDFS_HOME/build/tools" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HDFS_HOME/build/tools
|
||||||
|
fi
|
||||||
|
|
||||||
|
# for releases, add core hdfs jar & webapps to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_HDFS_HOME/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HDFS_HOME
|
||||||
|
fi
|
||||||
|
for f in $HADOOP_HDFS_HOME/hadoop-*-hdfs.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
# add libs to CLASSPATH
|
||||||
|
for f in $HADOOP_HDFS_HOME/lib/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
if $cygwin; then
|
||||||
|
CLASSPATH=`cygpath -p -w "$CLASSPATH"`
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
# included in all the hdfs scripts with source command
|
||||||
|
# should not be executed directly
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
#TODO: change the env variable when directory structure is changed
|
||||||
|
export HADOOP_CORE_HOME="${HADOOP_CORE_HOME:-$bin/..}"
|
||||||
|
#export HADOOP_CORE_HOME="${HADOOP_CORE_HOME:-$bin/../../core}"
|
||||||
|
|
||||||
|
if [ -d "${HADOOP_CORE_HOME}" ]; then
|
||||||
|
. "$HADOOP_CORE_HOME"/bin/hadoop-config.sh
|
||||||
|
else
|
||||||
|
echo "Hadoop core not found."
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,100 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. $bin/mapred-config.sh
|
||||||
|
|
||||||
|
function print_usage(){
|
||||||
|
echo "Usage: mapred [--config confdir] COMMAND"
|
||||||
|
echo " where COMMAND is one of:"
|
||||||
|
echo " mradmin run a Map-Reduce admin client"
|
||||||
|
echo " jobtracker run the MapReduce job Tracker node"
|
||||||
|
echo " tasktracker run a MapReduce task Tracker node"
|
||||||
|
echo " pipes run a Pipes job"
|
||||||
|
echo " job manipulate MapReduce jobs"
|
||||||
|
echo " queue get information regarding JobQueues"
|
||||||
|
echo ""
|
||||||
|
echo "Most commands print help when invoked w/o parameters."
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ $# = 0 ]; then
|
||||||
|
print_usage
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
|
||||||
|
COMMAND=$1
|
||||||
|
shift
|
||||||
|
|
||||||
|
if [ "$COMMAND" = "mradmin" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.mapred.tools.MRAdmin
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "jobtracker" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.mapred.JobTracker
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_JOBTRACKER_OPTS"
|
||||||
|
elif [ "$COMMAND" = "tasktracker" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.mapred.TaskTracker
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_TASKTRACKER_OPTS"
|
||||||
|
elif [ "$COMMAND" = "job" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.mapred.JobClient
|
||||||
|
elif [ "$COMMAND" = "queue" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.mapred.JobQueueClient
|
||||||
|
elif [ "$COMMAND" = "pipes" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.mapred.pipes.Submitter
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
elif [ "$COMMAND" = "sampler" ] ; then
|
||||||
|
CLASS=org.apache.hadoop.mapred.lib.InputSampler
|
||||||
|
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
|
||||||
|
else
|
||||||
|
echo $COMMAND - invalid command
|
||||||
|
print_usage
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
|
||||||
|
# for developers, add mapred classes to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_MAPRED_HOME/build/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_MAPRED_HOME/build/classes
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_MAPRED_HOME/build/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_MAPRED_HOME/build
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_MAPRED_HOME/build/test/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_MAPRED_HOME/build/test/classes
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_MAPRED_HOME/build/tools" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_MAPRED_HOME/build/tools
|
||||||
|
fi
|
||||||
|
|
||||||
|
# for releases, add core mapred jar & webapps to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_MAPRED_HOME/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_MAPRED_HOME
|
||||||
|
fi
|
||||||
|
for f in $HADOOP_MAPRED_HOME/hadoop-*-mapred.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
# add libs to CLASSPATH
|
||||||
|
for f in $HADOOP_MAPRED_HOME/lib/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
if $cygwin; then
|
||||||
|
CLASSPATH=`cygpath -p -w "$CLASSPATH"`
|
||||||
|
fi
|
||||||
|
export CLASSPATH
|
||||||
|
exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
# included in all the mapred scripts with source command
|
||||||
|
# should not be executed directly
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
#TODO: change the env variable when directory structure is changed
|
||||||
|
export HADOOP_CORE_HOME="${HADOOP_CORE_HOME:-$bin/..}"
|
||||||
|
#export HADOOP_CORE_HOME="${HADOOP_CORE_HOME:-$bin/../../core}"
|
||||||
|
|
||||||
|
if [ -d "${HADOOP_CORE_HOME}" ]; then
|
||||||
|
. "$HADOOP_CORE_HOME"/bin/hadoop-config.sh
|
||||||
|
else
|
||||||
|
echo "Hadoop core not found."
|
||||||
|
exit
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,99 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# The Hadoop record compiler
|
||||||
|
#
|
||||||
|
# Environment Variables
|
||||||
|
#
|
||||||
|
# JAVA_HOME The java implementation to use. Overrides JAVA_HOME.
|
||||||
|
#
|
||||||
|
# HADOOP_OPTS Extra Java runtime options.
|
||||||
|
#
|
||||||
|
# HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_HOME}/conf.
|
||||||
|
#
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hadoop-config.sh
|
||||||
|
|
||||||
|
if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then
|
||||||
|
. "${HADOOP_CONF_DIR}/hadoop-env.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# some Java parameters
|
||||||
|
if [ "$JAVA_HOME" != "" ]; then
|
||||||
|
#echo "run java in $JAVA_HOME"
|
||||||
|
JAVA_HOME=$JAVA_HOME
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$JAVA_HOME" = "" ]; then
|
||||||
|
echo "Error: JAVA_HOME is not set."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
JAVA=$JAVA_HOME/bin/java
|
||||||
|
JAVA_HEAP_MAX=-Xmx1000m
|
||||||
|
|
||||||
|
# CLASSPATH initially contains $HADOOP_CONF_DIR
|
||||||
|
CLASSPATH="${HADOOP_CONF_DIR}"
|
||||||
|
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
|
||||||
|
|
||||||
|
# for developers, add Hadoop classes to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_HOME/build/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/classes
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_HOME/build/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build
|
||||||
|
fi
|
||||||
|
if [ -d "$HADOOP_HOME/build/test/classes" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/test/classes
|
||||||
|
fi
|
||||||
|
|
||||||
|
# so that filenames w/ spaces are handled correctly in loops below
|
||||||
|
IFS=
|
||||||
|
|
||||||
|
# for releases, add core hadoop jar & webapps to CLASSPATH
|
||||||
|
if [ -d "$HADOOP_HOME/webapps" ]; then
|
||||||
|
CLASSPATH=${CLASSPATH}:$HADOOP_HOME
|
||||||
|
fi
|
||||||
|
for f in $HADOOP_HOME/hadoop-*-core.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
# add libs to CLASSPATH
|
||||||
|
for f in $HADOOP_HOME/lib/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
for f in $HADOOP_HOME/lib/jetty-ext/*.jar; do
|
||||||
|
CLASSPATH=${CLASSPATH}:$f;
|
||||||
|
done
|
||||||
|
|
||||||
|
# restore ordinary behaviour
|
||||||
|
unset IFS
|
||||||
|
|
||||||
|
CLASS='org.apache.hadoop.record.compiler.generated.Rcc'
|
||||||
|
|
||||||
|
# cygwin path translation
|
||||||
|
if expr `uname` : 'CYGWIN*' > /dev/null; then
|
||||||
|
CLASSPATH=`cygpath -p -w "$CLASSPATH"`
|
||||||
|
fi
|
||||||
|
|
||||||
|
# run it
|
||||||
|
exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Run a shell command on all slave hosts.
|
||||||
|
#
|
||||||
|
# Environment Variables
|
||||||
|
#
|
||||||
|
# HADOOP_SLAVES File naming remote hosts.
|
||||||
|
# Default is ${HADOOP_CONF_DIR}/slaves.
|
||||||
|
# HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_HOME}/conf.
|
||||||
|
# HADOOP_SLAVE_SLEEP Seconds to sleep between spawning remote commands.
|
||||||
|
# HADOOP_SSH_OPTS Options passed to ssh when running remote commands.
|
||||||
|
##
|
||||||
|
|
||||||
|
usage="Usage: slaves.sh [--config confdir] command..."
|
||||||
|
|
||||||
|
# if no args specified, show usage
|
||||||
|
if [ $# -le 0 ]; then
|
||||||
|
echo $usage
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hadoop-config.sh
|
||||||
|
|
||||||
|
# If the slaves file is specified in the command line,
|
||||||
|
# then it takes precedence over the definition in
|
||||||
|
# hadoop-env.sh. Save it here.
|
||||||
|
HOSTLIST=$HADOOP_SLAVES
|
||||||
|
|
||||||
|
if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then
|
||||||
|
. "${HADOOP_CONF_DIR}/hadoop-env.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$HOSTLIST" = "" ]; then
|
||||||
|
if [ "$HADOOP_SLAVES" = "" ]; then
|
||||||
|
export HOSTLIST="${HADOOP_CONF_DIR}/slaves"
|
||||||
|
else
|
||||||
|
export HOSTLIST="${HADOOP_SLAVES}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
|
||||||
|
ssh $HADOOP_SSH_OPTS $slave $"${@// /\\ }" \
|
||||||
|
2>&1 | sed "s/^/$slave: /" &
|
||||||
|
if [ "$HADOOP_SLAVE_SLEEP" != "" ]; then
|
||||||
|
sleep $HADOOP_SLAVE_SLEEP
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
wait
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Start all hadoop daemons. Run this on master node.
|
||||||
|
|
||||||
|
echo "This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh"
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hadoop-config.sh
|
||||||
|
|
||||||
|
# start hdfs daemons if hdfs is present
|
||||||
|
if [ -f "${HADOOP_HDFS_HOME}"/bin/start-dfs.sh ]; then
|
||||||
|
"${HADOOP_HDFS_HOME}"/bin/start-dfs.sh --config $HADOOP_CONF_DIR
|
||||||
|
fi
|
||||||
|
|
||||||
|
# start mapred daemons if mapred is present
|
||||||
|
if [ -f "${HADOOP_MAPRED_HOME}"/bin/start-mapred.sh ]; then
|
||||||
|
"${HADOOP_MAPRED_HOME}"/bin/start-mapred.sh --config $HADOOP_CONF_DIR
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hdfs-config.sh
|
||||||
|
|
||||||
|
# Start balancer daemon.
|
||||||
|
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs start balancer $@
|
||||||
|
|
@ -0,0 +1,52 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Start hadoop dfs daemons.
|
||||||
|
# Optinally upgrade or rollback dfs state.
|
||||||
|
# Run this on master node.
|
||||||
|
|
||||||
|
usage="Usage: start-dfs.sh [-upgrade|-rollback]"
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hdfs-config.sh
|
||||||
|
|
||||||
|
# get arguments
|
||||||
|
if [ $# -ge 1 ]; then
|
||||||
|
nameStartOpt=$1
|
||||||
|
shift
|
||||||
|
case $nameStartOpt in
|
||||||
|
(-upgrade)
|
||||||
|
;;
|
||||||
|
(-rollback)
|
||||||
|
dataStartOpt=$nameStartOpt
|
||||||
|
;;
|
||||||
|
(*)
|
||||||
|
echo $usage
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
|
||||||
|
# start dfs daemons
|
||||||
|
# start namenode after datanodes, to minimize time namenode is up w/o data
|
||||||
|
# note: datanodes will log connection errors until namenode starts
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs start namenode $nameStartOpt
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs start datanode $dataStartOpt
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --hosts masters --script "$bin"/hdfs start secondarynamenode
|
||||||
|
|
@ -0,0 +1,29 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Start hadoop map reduce daemons. Run this on master node.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. $bin/mapred-config.sh
|
||||||
|
|
||||||
|
# start mapred daemons
|
||||||
|
# start jobtracker first to minimize connection errors at startup
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/mapred start jobtracker
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script "$bin"/mapred start tasktracker
|
||||||
|
|
@ -0,0 +1,37 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Stop all hadoop daemons. Run this on master node.
|
||||||
|
|
||||||
|
echo "This script is Deprecated. Instead use stop-dfs.sh and stop-mapred.sh"
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hadoop-config.sh
|
||||||
|
|
||||||
|
# stop hdfs daemons if hdfs is present
|
||||||
|
if [ -f "${HADOOP_HDFS_HOME}"/bin/stop-dfs.sh ]; then
|
||||||
|
"${HADOOP_HDFS_HOME}"/bin/stop-dfs.sh --config $HADOOP_CONF_DIR
|
||||||
|
fi
|
||||||
|
|
||||||
|
# stop mapred daemons if mapred is present
|
||||||
|
if [ -f "${HADOOP_MAPRED_HOME}"/bin/stop-mapred.sh ]; then
|
||||||
|
"${HADOOP_MAPRED_HOME}"/bin/stop-mapred.sh --config $HADOOP_CONF_DIR
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
@ -0,0 +1,26 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hdfs-config.sh
|
||||||
|
|
||||||
|
# Stop balancer daemon.
|
||||||
|
# Run this on the machine where the balancer is running
|
||||||
|
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs stop balancer
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Stop hadoop DFS daemons. Run this on master node.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. "$bin"/hdfs-config.sh
|
||||||
|
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs stop namenode
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs stop datanode
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --hosts masters --script "$bin"/hdfs stop secondarynamenode
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
# Stop hadoop map reduce daemons. Run this on master node.
|
||||||
|
|
||||||
|
bin=`dirname "$0"`
|
||||||
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
. $bin/mapred-config.sh
|
||||||
|
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/mapred stop jobtracker
|
||||||
|
"$HADOOP_CORE_HOME"/bin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script "$bin"/mapred stop tasktracker
|
||||||
|
|
@ -0,0 +1,126 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
|
||||||
|
<!-- This is the configuration file for the resource manager in Hadoop. -->
|
||||||
|
<!-- You can configure various scheduling parameters related to queues. -->
|
||||||
|
<!-- The properties for a queue follow a naming convention,such as, -->
|
||||||
|
<!-- mapred.capacity-scheduler.queue.<queue-name>.property-name. -->
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.queue.default.capacity</name>
|
||||||
|
<value>100</value>
|
||||||
|
<description>Percentage of the number of slots in the cluster that are
|
||||||
|
to be available for jobs in this queue.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.queue.default.supports-priority</name>
|
||||||
|
<value>false</value>
|
||||||
|
<description>If true, priorities of jobs will be taken into
|
||||||
|
account in scheduling decisions.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.queue.default.minimum-user-limit-percent</name>
|
||||||
|
<value>100</value>
|
||||||
|
<description> Each queue enforces a limit on the percentage of resources
|
||||||
|
allocated to a user at any given time, if there is competition for them.
|
||||||
|
This user limit can vary between a minimum and maximum value. The former
|
||||||
|
depends on the number of users who have submitted jobs, and the latter is
|
||||||
|
set to this property value. For example, suppose the value of this
|
||||||
|
property is 25. If two users have submitted jobs to a queue, no single
|
||||||
|
user can use more than 50% of the queue resources. If a third user submits
|
||||||
|
a job, no single user can use more than 33% of the queue resources. With 4
|
||||||
|
or more users, no user can use more than 25% of the queue's resources. A
|
||||||
|
value of 100 implies no user limits are imposed.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.queue.default.maximum-initialized-jobs-per-user</name>
|
||||||
|
<value>2</value>
|
||||||
|
<description>The maximum number of jobs to be pre-initialized for a user
|
||||||
|
of the job queue.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<!-- The default configuration settings for the capacity task scheduler -->
|
||||||
|
<!-- The default values would be applied to all the queues which don't have -->
|
||||||
|
<!-- the appropriate property for the particular queue -->
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.default-supports-priority</name>
|
||||||
|
<value>false</value>
|
||||||
|
<description>If true, priorities of jobs will be taken into
|
||||||
|
account in scheduling decisions by default in a job queue.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.task.default-pmem-percentage-in-vmem</name>
|
||||||
|
<value>-1</value>
|
||||||
|
<description>A percentage (float) of the default VM limit for jobs
|
||||||
|
(mapred.task.default.maxvm). This is the default RAM task-limit
|
||||||
|
associated with a task. Unless overridden by a job's setting, this
|
||||||
|
number defines the RAM task-limit.
|
||||||
|
|
||||||
|
If this property is missing, or set to an invalid value, scheduling
|
||||||
|
based on physical memory, RAM, is disabled.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.task.limit.maxpmem</name>
|
||||||
|
<value>-1</value>
|
||||||
|
<description>Configuration that provides an upper limit on the maximum
|
||||||
|
physical memory that can be specified by a job. The job configuration
|
||||||
|
mapred.task.maxpmem should be less than this value. If not, the job will
|
||||||
|
be rejected by the scheduler.
|
||||||
|
|
||||||
|
If it is set to -1, scheduler will not consider physical memory for
|
||||||
|
scheduling even if virtual memory based scheduling is enabled(by setting
|
||||||
|
valid values for both mapred.task.default.maxvmem and
|
||||||
|
mapred.task.limit.maxvmem).
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.default-minimum-user-limit-percent</name>
|
||||||
|
<value>100</value>
|
||||||
|
<description>The percentage of the resources limited to a particular user
|
||||||
|
for the job queue at any given point of time by default.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.default-maximum-initialized-jobs-per-user</name>
|
||||||
|
<value>2</value>
|
||||||
|
<description>The maximum number of jobs to be pre-initialized for a user
|
||||||
|
of the job queue.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
|
||||||
|
<!-- Capacity scheduler Job Initialization configuration parameters -->
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.init-poll-interval</name>
|
||||||
|
<value>5000</value>
|
||||||
|
<description>The amount of time in miliseconds which is used to poll
|
||||||
|
the job queues for jobs to initialize.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
<property>
|
||||||
|
<name>mapred.capacity-scheduler.init-worker-threads</name>
|
||||||
|
<value>5</value>
|
||||||
|
<description>Number of worker threads which would be used by
|
||||||
|
Initialization poller to initialize jobs in a set of queue.
|
||||||
|
If number mentioned in property is equal to number of job queues
|
||||||
|
then a single thread would initialize jobs in a queue. If lesser
|
||||||
|
then a thread would get a set of queues assigned. If the number
|
||||||
|
is greater then number of threads would be equal to number of
|
||||||
|
job queues.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
|
||||||
|
<xsl:output method="html"/>
|
||||||
|
<xsl:template match="configuration">
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<table border="1">
|
||||||
|
<tr>
|
||||||
|
<td>name</td>
|
||||||
|
<td>value</td>
|
||||||
|
<td>description</td>
|
||||||
|
</tr>
|
||||||
|
<xsl:for-each select="property">
|
||||||
|
<tr>
|
||||||
|
<td><a name="{name}"><xsl:value-of select="name"/></a></td>
|
||||||
|
<td><xsl:value-of select="value"/></td>
|
||||||
|
<td><xsl:value-of select="description"/></td>
|
||||||
|
</tr>
|
||||||
|
</xsl:for-each>
|
||||||
|
</table>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
</xsl:template>
|
||||||
|
</xsl:stylesheet>
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||||
|
|
||||||
|
<!-- Put site-specific property overrides in this file. -->
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1,54 @@
|
||||||
|
# Set Hadoop-specific environment variables here.
|
||||||
|
|
||||||
|
# The only required environment variable is JAVA_HOME. All others are
|
||||||
|
# optional. When running a distributed configuration it is best to
|
||||||
|
# set JAVA_HOME in this file, so that it is correctly defined on
|
||||||
|
# remote nodes.
|
||||||
|
|
||||||
|
# The java implementation to use. Required.
|
||||||
|
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
|
||||||
|
|
||||||
|
# Extra Java CLASSPATH elements. Optional.
|
||||||
|
# export HADOOP_CLASSPATH=
|
||||||
|
|
||||||
|
# The maximum amount of heap to use, in MB. Default is 1000.
|
||||||
|
# export HADOOP_HEAPSIZE=2000
|
||||||
|
|
||||||
|
# Extra Java runtime options. Empty by default.
|
||||||
|
# export HADOOP_OPTS=-server
|
||||||
|
|
||||||
|
# Command specific options appended to HADOOP_OPTS when specified
|
||||||
|
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
|
||||||
|
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
|
||||||
|
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
|
||||||
|
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
|
||||||
|
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
|
||||||
|
# export HADOOP_TASKTRACKER_OPTS=
|
||||||
|
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
|
||||||
|
# export HADOOP_CLIENT_OPTS
|
||||||
|
|
||||||
|
# Extra ssh options. Empty by default.
|
||||||
|
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
|
||||||
|
|
||||||
|
# Where log files are stored. $HADOOP_HOME/logs by default.
|
||||||
|
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
|
||||||
|
|
||||||
|
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
|
||||||
|
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
|
||||||
|
|
||||||
|
# host:path where hadoop code should be rsync'd from. Unset by default.
|
||||||
|
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
|
||||||
|
|
||||||
|
# Seconds to sleep between slave commands. Unset by default. This
|
||||||
|
# can be useful in large clusters, where, e.g., slave rsyncs can
|
||||||
|
# otherwise arrive faster than the master can service them.
|
||||||
|
# export HADOOP_SLAVE_SLEEP=0.1
|
||||||
|
|
||||||
|
# The directory where pid files are stored. /tmp by default.
|
||||||
|
# export HADOOP_PID_DIR=/var/hadoop/pids
|
||||||
|
|
||||||
|
# A string representing this instance of hadoop. $USER by default.
|
||||||
|
# export HADOOP_IDENT_STRING=$USER
|
||||||
|
|
||||||
|
# The scheduling priority for daemon processes. See 'man nice'.
|
||||||
|
# export HADOOP_NICENESS=10
|
||||||
|
|
@ -0,0 +1,53 @@
|
||||||
|
# Configuration of the "dfs" context for null
|
||||||
|
dfs.class=org.apache.hadoop.metrics.spi.NullContext
|
||||||
|
|
||||||
|
# Configuration of the "dfs" context for file
|
||||||
|
#dfs.class=org.apache.hadoop.metrics.file.FileContext
|
||||||
|
#dfs.period=10
|
||||||
|
#dfs.fileName=/tmp/dfsmetrics.log
|
||||||
|
|
||||||
|
# Configuration of the "dfs" context for ganglia
|
||||||
|
# dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||||
|
# dfs.period=10
|
||||||
|
# dfs.servers=localhost:8649
|
||||||
|
|
||||||
|
|
||||||
|
# Configuration of the "mapred" context for null
|
||||||
|
mapred.class=org.apache.hadoop.metrics.spi.NullContext
|
||||||
|
|
||||||
|
# Configuration of the "mapred" context for file
|
||||||
|
#mapred.class=org.apache.hadoop.metrics.file.FileContext
|
||||||
|
#mapred.period=10
|
||||||
|
#mapred.fileName=/tmp/mrmetrics.log
|
||||||
|
|
||||||
|
# Configuration of the "mapred" context for ganglia
|
||||||
|
# mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||||
|
# mapred.period=10
|
||||||
|
# mapred.servers=localhost:8649
|
||||||
|
|
||||||
|
|
||||||
|
# Configuration of the "jvm" context for null
|
||||||
|
jvm.class=org.apache.hadoop.metrics.spi.NullContext
|
||||||
|
|
||||||
|
# Configuration of the "jvm" context for file
|
||||||
|
#jvm.class=org.apache.hadoop.metrics.file.FileContext
|
||||||
|
#jvm.period=10
|
||||||
|
#jvm.fileName=/tmp/jvmmetrics.log
|
||||||
|
|
||||||
|
# Configuration of the "jvm" context for ganglia
|
||||||
|
# jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||||
|
# jvm.period=10
|
||||||
|
# jvm.servers=localhost:8649
|
||||||
|
|
||||||
|
# Configuration of the "rpc" context for null
|
||||||
|
rpc.class=org.apache.hadoop.metrics.spi.NullContext
|
||||||
|
|
||||||
|
# Configuration of the "rpc" context for file
|
||||||
|
#rpc.class=org.apache.hadoop.metrics.file.FileContext
|
||||||
|
#rpc.period=10
|
||||||
|
#rpc.fileName=/tmp/rpcmetrics.log
|
||||||
|
|
||||||
|
# Configuration of the "rpc" context for ganglia
|
||||||
|
# rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||||
|
# rpc.period=10
|
||||||
|
# rpc.servers=localhost:8649
|
||||||
|
|
@ -0,0 +1,97 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||||
|
|
||||||
|
<!-- Put site-specific property overrides in this file. -->
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
<property>
|
||||||
|
<name>security.client.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for ClientProtocol, which is used by user code
|
||||||
|
via the DistributedFileSystem.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.client.datanode.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for ClientDatanodeProtocol, the client-to-datanode protocol
|
||||||
|
for block recovery.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.datanode.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for DatanodeProtocol, which is used by datanodes to
|
||||||
|
communicate with the namenode.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.inter.datanode.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for InterDatanodeProtocol, the inter-datanode protocol
|
||||||
|
for updating generation timestamp.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.namenode.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for NamenodeProtocol, the protocol used by the secondary
|
||||||
|
namenode to communicate with the namenode.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.inter.tracker.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for InterTrackerProtocol, used by the tasktrackers to
|
||||||
|
communicate with the jobtracker.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.job.submission.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for JobSubmissionProtocol, used by job clients to
|
||||||
|
communciate with the jobtracker for job submission, querying job status etc.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.task.umbilical.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for TaskUmbilicalProtocol, used by the map and reduce
|
||||||
|
tasks to communicate with the parent tasktracker.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>security.refresh.policy.protocol.acl</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description>ACL for RefreshAuthorizationPolicyProtocol, used by the
|
||||||
|
dfsadmin and mradmin commands to refresh the security policy in-effect.
|
||||||
|
The ACL is a comma-separated list of user and group names. The user and
|
||||||
|
group list is separated by a blank. For e.g. "alice,bob users,wheel".
|
||||||
|
A special value of "*" means all users are allowed.</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||||
|
|
||||||
|
<!-- Put site-specific property overrides in this file. -->
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1,94 @@
|
||||||
|
# Define some default values that can be overridden by system properties
|
||||||
|
hadoop.root.logger=INFO,console
|
||||||
|
hadoop.log.dir=.
|
||||||
|
hadoop.log.file=hadoop.log
|
||||||
|
|
||||||
|
# Define the root logger to the system property "hadoop.root.logger".
|
||||||
|
log4j.rootLogger=${hadoop.root.logger}, EventCounter
|
||||||
|
|
||||||
|
# Logging Threshold
|
||||||
|
log4j.threshhold=ALL
|
||||||
|
|
||||||
|
#
|
||||||
|
# Daily Rolling File Appender
|
||||||
|
#
|
||||||
|
|
||||||
|
log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
|
||||||
|
log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}
|
||||||
|
|
||||||
|
# Rollver at midnight
|
||||||
|
log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
|
||||||
|
|
||||||
|
# 30-day backup
|
||||||
|
#log4j.appender.DRFA.MaxBackupIndex=30
|
||||||
|
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
|
||||||
|
|
||||||
|
# Pattern format: Date LogLevel LoggerName LogMessage
|
||||||
|
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
|
||||||
|
# Debugging Pattern format
|
||||||
|
#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
|
||||||
|
|
||||||
|
|
||||||
|
#
|
||||||
|
# console
|
||||||
|
# Add "console" to rootlogger above if you want to use this
|
||||||
|
#
|
||||||
|
|
||||||
|
log4j.appender.console=org.apache.log4j.ConsoleAppender
|
||||||
|
log4j.appender.console.target=System.err
|
||||||
|
log4j.appender.console.layout=org.apache.log4j.PatternLayout
|
||||||
|
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
|
||||||
|
|
||||||
|
#
|
||||||
|
# TaskLog Appender
|
||||||
|
#
|
||||||
|
|
||||||
|
#Default values
|
||||||
|
hadoop.tasklog.taskid=null
|
||||||
|
hadoop.tasklog.noKeepSplits=4
|
||||||
|
hadoop.tasklog.totalLogFileSize=100
|
||||||
|
hadoop.tasklog.purgeLogSplits=true
|
||||||
|
hadoop.tasklog.logsRetainHours=12
|
||||||
|
|
||||||
|
log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender
|
||||||
|
log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
|
||||||
|
log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize}
|
||||||
|
|
||||||
|
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
|
||||||
|
log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
|
||||||
|
|
||||||
|
#
|
||||||
|
# Rolling File Appender
|
||||||
|
#
|
||||||
|
|
||||||
|
#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
|
||||||
|
#log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
|
||||||
|
|
||||||
|
# Logfile size and and 30-day backups
|
||||||
|
#log4j.appender.RFA.MaxFileSize=1MB
|
||||||
|
#log4j.appender.RFA.MaxBackupIndex=30
|
||||||
|
|
||||||
|
#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
|
||||||
|
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n
|
||||||
|
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
|
||||||
|
|
||||||
|
#
|
||||||
|
# FSNamesystem Audit logging
|
||||||
|
# All audit events are logged at INFO level
|
||||||
|
#
|
||||||
|
log4j.logger.org.apache.hadoop.fs.FSNamesystem.audit=WARN
|
||||||
|
|
||||||
|
# Custom Logging levels
|
||||||
|
|
||||||
|
#log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG
|
||||||
|
#log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG
|
||||||
|
#log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG
|
||||||
|
|
||||||
|
# Jets3t library
|
||||||
|
log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR
|
||||||
|
|
||||||
|
#
|
||||||
|
# Event Counter Appender
|
||||||
|
# Sends counts of logging messages at different severity levels to Hadoop Metrics.
|
||||||
|
#
|
||||||
|
log4j.appender.EventCounter=org.apache.hadoop.metrics.jvm.EventCounter
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||||
|
|
||||||
|
<!-- This is a template file for queue acls configuration properties -->
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.queue.default.acl-submit-job</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description> Comma separated list of user and group names that are allowed
|
||||||
|
to submit jobs to the 'default' queue. The user list and the group list
|
||||||
|
are separated by a blank. For e.g. alice,bob group1,group2.
|
||||||
|
If set to the special value '*', it means all users are allowed to
|
||||||
|
submit jobs.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>mapred.queue.default.acl-administer-jobs</name>
|
||||||
|
<value>*</value>
|
||||||
|
<description> Comma separated list of user and group names that are allowed
|
||||||
|
to delete jobs or modify job's priority for jobs not owned by the current
|
||||||
|
user in the 'default' queue. The user list and the group list
|
||||||
|
are separated by a blank. For e.g. alice,bob group1,group2.
|
||||||
|
If set to the special value '*', it means all users are allowed to do
|
||||||
|
this operation.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||||
|
|
||||||
|
<!-- Put site-specific property overrides in this file. -->
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
localhost
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
localhost
|
||||||
|
|
@ -0,0 +1,57 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.client.truststore.location</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Truststore to be used by clients like distcp. Must be
|
||||||
|
specified.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.client.truststore.password</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Optional. Default value is "".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.client.truststore.type</name>
|
||||||
|
<value>jks</value>
|
||||||
|
<description>Optional. Default value is "jks".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.client.keystore.location</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Keystore to be used by clients like distcp. Must be
|
||||||
|
specified.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.client.keystore.password</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Optional. Default value is "".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.client.keystore.keypassword</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Optional. Default value is "".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.client.keystore.type</name>
|
||||||
|
<value>jks</value>
|
||||||
|
<description>Optional. Default value is "jks".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||||
|
|
||||||
|
<configuration>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.server.truststore.location</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Truststore to be used by NN and DN. Must be specified.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.server.truststore.password</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Optional. Default value is "".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.server.truststore.type</name>
|
||||||
|
<value>jks</value>
|
||||||
|
<description>Optional. Default value is "jks".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.server.keystore.location</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Keystore to be used by NN and DN. Must be specified.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.server.keystore.password</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Must be specified.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.server.keystore.keypassword</name>
|
||||||
|
<value></value>
|
||||||
|
<description>Must be specified.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>ssl.server.keystore.type</name>
|
||||||
|
<value>jks</value>
|
||||||
|
<description>Optional. Default value is "jks".
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
mapred.local.dir=#configured value of hadoop.tmp.dir it can be a list of paths comma seperated
|
||||||
|
hadoop.pid.dir=#configured HADOOP_PID_DIR
|
||||||
|
hadoop.indent.str=#configured HADOOP_IDENT_STR
|
||||||
|
|
@ -0,0 +1,264 @@
|
||||||
|
<?xml version="1.0" ?>
|
||||||
|
|
||||||
|
<!--
|
||||||
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
contributor license agreements. See the NOTICE file distributed with
|
||||||
|
this work for additional information regarding copyright ownership.
|
||||||
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
(the "License"); you may not use this file except in compliance with
|
||||||
|
the License. You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
|
<ivy-module version="1.0">
|
||||||
|
<info organisation="org.apache.hadoop" module="${ant.project.name}" revision="${version}">
|
||||||
|
<license name="Apache 2.0"/>
|
||||||
|
<ivyauthor name="Apache Hadoop Team" url="http://hadoop.apache.org"/>
|
||||||
|
<description>
|
||||||
|
Hadoop Core
|
||||||
|
</description>
|
||||||
|
</info>
|
||||||
|
<configurations defaultconfmapping="default">
|
||||||
|
<!--these match the Maven configurations-->
|
||||||
|
<conf name="default" extends="master,runtime"/>
|
||||||
|
<conf name="master" description="contains the artifact but no dependencies"/>
|
||||||
|
<conf name="runtime" description="runtime but not the artifact"
|
||||||
|
extends="client,server,s3-server,kfs"/>
|
||||||
|
|
||||||
|
<conf name="mandatory" description="contains the critical dependencies"
|
||||||
|
extends="commons-logging,log4j"/>
|
||||||
|
|
||||||
|
<!--
|
||||||
|
These public configurations contain the core dependencies for running hadoop client or server.
|
||||||
|
The server is effectively a superset of the client.
|
||||||
|
-->
|
||||||
|
<conf name="client" description="client-side dependencies"
|
||||||
|
extends="mandatory,httpclient"/>
|
||||||
|
<conf name="server" description="server-side dependencies"
|
||||||
|
extends="client"/>
|
||||||
|
<conf name="s3-client" description="dependencies for working with S3/EC2 infrastructure"
|
||||||
|
extends="client"/>
|
||||||
|
<conf name="s3-server" description="dependencies for running on S3/EC2 infrastructure"
|
||||||
|
extends="s3-client,server"/>
|
||||||
|
<conf name="kfs" description="dependencies for KFS file system support"/>
|
||||||
|
<conf name="ftp" description="dependencies for workign with FTP filesytems"
|
||||||
|
extends="mandatory"/>
|
||||||
|
<conf name="jetty" description="Jetty provides the in-VM HTTP daemon" extends="commons-logging"/>
|
||||||
|
|
||||||
|
<!--Private configurations. -->
|
||||||
|
|
||||||
|
<conf name="common" visibility="private" extends="runtime,mandatory,httpclient,ftp,jetty"
|
||||||
|
description="common artifacts"/>
|
||||||
|
<conf name="javadoc" visibility="private" description="artiracts required while performing doc generation"
|
||||||
|
extends="common,mandatory,jetty,lucene"/>
|
||||||
|
<!--Testing pulls in everything-->
|
||||||
|
<conf name="test" extends="common,default,s3-server,kfs" visibility="private"
|
||||||
|
description="the classpath needed to run tests"/>
|
||||||
|
<conf name="releaseaudit" visibility="private"
|
||||||
|
description="Artifacts required for releaseaudit target"/>
|
||||||
|
|
||||||
|
<conf name="commons-logging" visibility="private"/>
|
||||||
|
<conf name="httpclient" visibility="private" extends="commons-logging"/>
|
||||||
|
<conf name="log4j" visibility="private"/>
|
||||||
|
<conf name="lucene" visibility="private"/>
|
||||||
|
<conf name="jdiff" visibility="private" extends="log4j,s3-client,jetty,server"/>
|
||||||
|
<conf name="checkstyle" visibility="private"/>
|
||||||
|
|
||||||
|
</configurations>
|
||||||
|
|
||||||
|
<publications>
|
||||||
|
<!--get the artifact from our module name-->
|
||||||
|
<artifact conf="master"/>
|
||||||
|
</publications>
|
||||||
|
<dependencies>
|
||||||
|
|
||||||
|
<!--used client side-->
|
||||||
|
<!-- <dependency org="commons-cli"
|
||||||
|
name="commons-cli"
|
||||||
|
rev="${commons-cli.version}"
|
||||||
|
conf="client->default"/> -->
|
||||||
|
|
||||||
|
<dependency org="checkstyle"
|
||||||
|
name="checkstyle"
|
||||||
|
rev="${checkstyle.version}"
|
||||||
|
conf="checkstyle->default"/>
|
||||||
|
<dependency org="jdiff"
|
||||||
|
name="jdiff"
|
||||||
|
rev="${jdiff.version}"
|
||||||
|
conf="jdiff->default"/>
|
||||||
|
<dependency org="xerces"
|
||||||
|
name="xerces"
|
||||||
|
rev="${xerces.version}"
|
||||||
|
conf="jdiff->default">
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency org="xmlenc"
|
||||||
|
name="xmlenc"
|
||||||
|
rev="${xmlenc.version}"
|
||||||
|
conf="server->default"/>
|
||||||
|
|
||||||
|
<!--Configuration: httpclient-->
|
||||||
|
|
||||||
|
<!--
|
||||||
|
commons-httpclient asks for too many files.
|
||||||
|
All it needs is commons-codec and commons-logging JARs
|
||||||
|
-->
|
||||||
|
<dependency org="commons-httpclient"
|
||||||
|
name="commons-httpclient"
|
||||||
|
rev="${commons-httpclient.version}"
|
||||||
|
conf="httpclient->master">
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency org="commons-codec"
|
||||||
|
name="commons-codec"
|
||||||
|
rev="${commons-codec.version}"
|
||||||
|
conf="httpclient->default"/>
|
||||||
|
|
||||||
|
<dependency org="commons-net"
|
||||||
|
name="commons-net"
|
||||||
|
rev="${commons-net.version}"
|
||||||
|
conf="ftp->default"/>
|
||||||
|
|
||||||
|
<!--Configuration: Jetty -->
|
||||||
|
|
||||||
|
<!-- <dependency org="javax.servlet"
|
||||||
|
name="servlet-api"
|
||||||
|
rev="${servlet-api.version}"
|
||||||
|
conf="jetty->master"/> -->
|
||||||
|
<dependency org="org.mortbay.jetty"
|
||||||
|
name="jetty"
|
||||||
|
rev="${jetty.version}"
|
||||||
|
conf="jetty->master"/>
|
||||||
|
<dependency org="org.mortbay.jetty"
|
||||||
|
name="jetty-util"
|
||||||
|
rev="${jetty-util.version}"
|
||||||
|
conf="jetty->master"/>
|
||||||
|
|
||||||
|
<dependency org="tomcat"
|
||||||
|
name="jasper-runtime"
|
||||||
|
rev="${jasper.version}"
|
||||||
|
conf="jetty->master"/>
|
||||||
|
<dependency org="tomcat"
|
||||||
|
name="jasper-compiler"
|
||||||
|
rev="${jasper.version}"
|
||||||
|
conf="jetty->master"/>
|
||||||
|
<dependency org="org.mortbay.jetty"
|
||||||
|
name="jsp-api-2.1"
|
||||||
|
rev="${jetty.version}"
|
||||||
|
conf="jetty->master"/>
|
||||||
|
<dependency org="org.mortbay.jetty"
|
||||||
|
name="jsp-2.1"
|
||||||
|
rev="${jetty.version}"
|
||||||
|
conf="jetty->master"/>
|
||||||
|
<dependency org="commons-el"
|
||||||
|
name="commons-el"
|
||||||
|
rev="${commons-el.version}"
|
||||||
|
conf="jetty->master"/>
|
||||||
|
|
||||||
|
|
||||||
|
<!--Configuration: commons-logging -->
|
||||||
|
|
||||||
|
<!--it is essential that only the master JAR of commons logging
|
||||||
|
is pulled in, as its dependencies are usually a mess, including things
|
||||||
|
like out of date servlet APIs, bits of Avalon, etc.
|
||||||
|
-->
|
||||||
|
<dependency org="commons-logging"
|
||||||
|
name="commons-logging"
|
||||||
|
rev="${commons-logging.version}"
|
||||||
|
conf="commons-logging->master"/>
|
||||||
|
|
||||||
|
|
||||||
|
<!--Configuration: commons-logging -->
|
||||||
|
|
||||||
|
<!--log4J is not optional until commons-logging.properties is stripped out of the JAR -->
|
||||||
|
<dependency org="log4j"
|
||||||
|
name="log4j"
|
||||||
|
rev="${log4j.version}"
|
||||||
|
conf="log4j->master"/>
|
||||||
|
|
||||||
|
<!--Configuration: s3-client -->
|
||||||
|
<!--there are two jets3t projects in the repository; this one goes up to 0.6 and
|
||||||
|
is assumed to be the live one-->
|
||||||
|
<dependency org="net.java.dev.jets3t"
|
||||||
|
name="jets3t"
|
||||||
|
rev="${jets3t.version}"
|
||||||
|
conf="s3-client->master"/>
|
||||||
|
<dependency org="commons-net"
|
||||||
|
name="commons-net"
|
||||||
|
rev="${commons-net.version}"
|
||||||
|
conf="s3-client->master"/>
|
||||||
|
<dependency org="org.mortbay.jetty"
|
||||||
|
name="servlet-api-2.5"
|
||||||
|
rev="${servlet-api-2.5.version}"
|
||||||
|
conf="s3-client->master"/>
|
||||||
|
|
||||||
|
<!--Configuration: kfs -->
|
||||||
|
|
||||||
|
<!-- This is not in the repository
|
||||||
|
<dependency org="org.kosmix"
|
||||||
|
name="kfs"
|
||||||
|
rev="${kfs.version}"
|
||||||
|
conf="kfs->default"/>-->
|
||||||
|
|
||||||
|
<!--Configuration: test -->
|
||||||
|
|
||||||
|
<!--artifacts needed for testing -->
|
||||||
|
<dependency org="junit"
|
||||||
|
name="junit"
|
||||||
|
rev="${junit.version}"
|
||||||
|
conf="common->default"/>
|
||||||
|
<dependency org="com.google.code.p.arat"
|
||||||
|
name="rat-lib"
|
||||||
|
rev="${rats-lib.version}"
|
||||||
|
conf="releaseaudit->default"/>
|
||||||
|
<dependency org="commons-lang"
|
||||||
|
name="commons-lang"
|
||||||
|
rev="${commons-lang.version}"
|
||||||
|
conf="releaseaudit->default"/>
|
||||||
|
<dependency org="commons-collections"
|
||||||
|
name="commons-collections"
|
||||||
|
rev="${commons-collections.version}"
|
||||||
|
conf="releaseaudit->default"/>
|
||||||
|
<!--<dependency org="hsqldb"
|
||||||
|
name="hsqldb"
|
||||||
|
rev="${hsqldb.version}"
|
||||||
|
conf="common->default"/>
|
||||||
|
<dependency org="lucene"
|
||||||
|
name="lucene"
|
||||||
|
rev="${lucene.version}"
|
||||||
|
conf="javadoc->default"/> -->
|
||||||
|
<dependency org="org.apache.lucene"
|
||||||
|
name="lucene-core"
|
||||||
|
rev="${lucene-core.version}"
|
||||||
|
conf="javadoc->default"/>
|
||||||
|
<dependency org="commons-logging"
|
||||||
|
name="commons-logging-api"
|
||||||
|
rev="${commons-logging-api.version}"
|
||||||
|
conf="common->default"/>
|
||||||
|
<dependency org="org.slf4j"
|
||||||
|
name="slf4j-api"
|
||||||
|
rev="${slf4j-api.version}"
|
||||||
|
conf="common->master"/>
|
||||||
|
<dependency org="org.eclipse.jdt"
|
||||||
|
name="core"
|
||||||
|
rev="${core.version}"
|
||||||
|
conf="common->master"/>
|
||||||
|
<dependency org="oro"
|
||||||
|
name="oro"
|
||||||
|
rev="${oro.version}"
|
||||||
|
conf="common->default"/>
|
||||||
|
<dependency org="org.slf4j"
|
||||||
|
name="slf4j-log4j12"
|
||||||
|
rev="${slf4j-log4j12.version}"
|
||||||
|
conf="common->master">
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
|
||||||
|
</ivy-module>
|
||||||
|
|
@ -0,0 +1,257 @@
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
|
||||||
|
|
||||||
|
<!--
|
||||||
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
contributor license agreements. See the NOTICE file distributed with
|
||||||
|
this work for additional information regarding copyright ownership.
|
||||||
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
(the "License"); you may not use this file except in compliance with
|
||||||
|
the License. You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
<groupId>org.apache.hadoop</groupId>
|
||||||
|
<artifactId>hadoop-core</artifactId>
|
||||||
|
<packaging>jar</packaging>
|
||||||
|
<version>${hadoop.version}</version>
|
||||||
|
<description>
|
||||||
|
Hadoop is the distributed computing framework of Apache; hadoop-core contains
|
||||||
|
the filesystem, job tracker and map/reduce modules
|
||||||
|
</description>
|
||||||
|
<licenses>
|
||||||
|
<license>
|
||||||
|
<name>Apache License, Version 2.0</name>
|
||||||
|
<url>http://apache.org/licenses/LICENSE-2.0</url>
|
||||||
|
</license>
|
||||||
|
</licenses>
|
||||||
|
<dependencies>
|
||||||
|
|
||||||
|
|
||||||
|
<!-- always include commons-logging and log4J -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>commons-logging</groupId>
|
||||||
|
<artifactId>commons-logging</artifactId>
|
||||||
|
<version>${commons-logging.version}</version>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>avalon-framework</groupId>
|
||||||
|
<artifactId>avalon-framework</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>javax.servlet</groupId>
|
||||||
|
<artifactId>servlet-api</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>junit</groupId>
|
||||||
|
<artifactId>junit</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>logkit</groupId>
|
||||||
|
<artifactId>logkit</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>log4j</groupId>
|
||||||
|
<artifactId>log4j</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>log4j</groupId>
|
||||||
|
<artifactId>log4j</artifactId>
|
||||||
|
<version>${log4j.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>javax.mail</groupId>
|
||||||
|
<artifactId>mail</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>javax.jms</groupId>
|
||||||
|
<artifactId>jms</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>com.sun.jdmk</groupId>
|
||||||
|
<artifactId>jmxtools</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>com.sun.jmx</groupId>
|
||||||
|
<artifactId>jmxri</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!--SLF4J is a JAR-based dependency; this POM binds it to log4J-->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
<artifactId>slf4j-api</artifactId>
|
||||||
|
<version>${slf4j-api.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
<artifactId>slf4j-log4j12</artifactId>
|
||||||
|
<version>${slf4j-log4j12.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>log4j</groupId>
|
||||||
|
<artifactId>log4j</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!--Httpclient and its components are optional-->
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>commons-httpclient</groupId>
|
||||||
|
<artifactId>commons-httpclient</artifactId>
|
||||||
|
<version>3.1</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>commons-logging</groupId>
|
||||||
|
<artifactId>commons-logging</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>junit</groupId>
|
||||||
|
<artifactId>junit</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>commons-codec</groupId>
|
||||||
|
<artifactId>commons-codec</artifactId>
|
||||||
|
<version>1.3</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!--CLI is needed to scan the command line, but only the 1.0 branch is released -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>commons-cli</groupId>
|
||||||
|
<artifactId>commons-cli</artifactId>
|
||||||
|
<version>2.0-20070823</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
|
||||||
|
<!-- this is used for the ftp:// filesystem-->
|
||||||
|
<dependency>
|
||||||
|
<groupId>commons-net</groupId>
|
||||||
|
<artifactId>commons-net</artifactId>
|
||||||
|
<version>1.4.1</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- Jetty is used to serve up the application. It is marked as optional because
|
||||||
|
clients do not need it. All server-side deployments will need
|
||||||
|
all of these files.-->
|
||||||
|
<dependency>
|
||||||
|
<groupId>javax.servlet</groupId>
|
||||||
|
<artifactId>servlet-api</artifactId>
|
||||||
|
<version>${servlet-api.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>jetty</groupId>
|
||||||
|
<artifactId>org.mortbay.jetty</artifactId>
|
||||||
|
<version>${jetty.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
|
||||||
|
<!--JSP support -->
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.mortbay.jetty</groupId>
|
||||||
|
<artifactId>jsp-2.1</artifactId>
|
||||||
|
<version>${jetty.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.mortbay.jetty</groupId>
|
||||||
|
<artifactId>jsp-api-2.1</artifactId>
|
||||||
|
<version>${jetty.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>commons-el</groupId>
|
||||||
|
<artifactId>commons-el</artifactId>
|
||||||
|
<version>${commons-el.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
|
||||||
|
<!--JSPC assistance-->
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.eclipse.jdt</groupId>
|
||||||
|
<artifactId>core</artifactId>
|
||||||
|
<version>${core.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.ant</groupId>
|
||||||
|
<artifactId>ant</artifactId>
|
||||||
|
<version>${apacheant.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- JetS3t is a client library for S3.
|
||||||
|
-It is only needed if you want to work with S3 filesystems
|
||||||
|
-It pulls in commons-logging 1.1.1 and does not exclude all the cruft that comes with it.
|
||||||
|
By excluding it we stay in control of versions and dependencies
|
||||||
|
-->
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>net.java.dev.jets3t</groupId>
|
||||||
|
<artifactId>jets3t</artifactId>
|
||||||
|
<version>${jets3t.version}</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>commons-logging</groupId>
|
||||||
|
<artifactId>commons-logging</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>junit</groupId>
|
||||||
|
<artifactId>junit</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!--Kosmos filesystem
|
||||||
|
http://kosmosfs.sourceforge.net/
|
||||||
|
This is not in the central repository
|
||||||
|
-->
|
||||||
|
<!--
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.kosmix</groupId>
|
||||||
|
<artifactId>kfs</artifactId>
|
||||||
|
<version>0.1</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
-->
|
||||||
|
|
||||||
|
<!--
|
||||||
|
http://xmlenc.sourceforge.net/
|
||||||
|
"The xmlenc library is a fast stream-based XML output library for Java."
|
||||||
|
-->
|
||||||
|
<dependency>
|
||||||
|
<groupId>xmlenc</groupId>
|
||||||
|
<artifactId>xmlenc</artifactId>
|
||||||
|
<version>0.52</version>
|
||||||
|
<scope>optional</scope>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
</project>
|
||||||
|
|
@ -0,0 +1,81 @@
|
||||||
|
<ivysettings>
|
||||||
|
|
||||||
|
<!--
|
||||||
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
contributor license agreements. See the NOTICE file distributed with
|
||||||
|
this work for additional information regarding copyright ownership.
|
||||||
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
(the "License"); you may not use this file except in compliance with
|
||||||
|
the License. You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
|
<!--
|
||||||
|
see http://www.jayasoft.org/ivy/doc/configuration
|
||||||
|
-->
|
||||||
|
<!-- you can override this property to use mirrors
|
||||||
|
http://repo1.maven.org/maven2/
|
||||||
|
http://mirrors.dotsrc.org/maven2
|
||||||
|
http://ftp.ggi-project.org/pub/packages/maven2
|
||||||
|
http://mirrors.sunsite.dk/maven2
|
||||||
|
http://public.planetmirror.com/pub/maven2
|
||||||
|
http://ibiblio.lsu.edu/main/pub/packages/maven2
|
||||||
|
http://www.ibiblio.net/pub/packages/maven2
|
||||||
|
-->
|
||||||
|
<property name="repo.maven.org"
|
||||||
|
value="http://repo1.maven.org/maven2/"
|
||||||
|
override="false"/>
|
||||||
|
<property name="snapshot.apache.org"
|
||||||
|
value="http://people.apache.org/repo/m2-snapshot-repository/"
|
||||||
|
override="false"/>
|
||||||
|
<property name="maven2.pattern"
|
||||||
|
value="[organisation]/[module]/[revision]/[module]-[revision]"/>
|
||||||
|
<property name="maven2.pattern.ext"
|
||||||
|
value="${maven2.pattern}.[ext]"/>
|
||||||
|
<!-- pull in the local repository -->
|
||||||
|
<include url="${ivy.default.conf.dir}/ivyconf-local.xml"/>
|
||||||
|
<settings defaultResolver="default"/>
|
||||||
|
<resolvers>
|
||||||
|
<ibiblio name="maven2"
|
||||||
|
root="${repo.maven.org}"
|
||||||
|
pattern="${maven2.pattern.ext}"
|
||||||
|
m2compatible="true"
|
||||||
|
/>
|
||||||
|
<ibiblio name="apache-snapshot"
|
||||||
|
root="${snapshot.apache.org}"
|
||||||
|
pattern="${maven2.pattern.ext}"
|
||||||
|
m2compatible="true"
|
||||||
|
/>
|
||||||
|
<chain name="default" dual="true">
|
||||||
|
<resolver ref="local"/>
|
||||||
|
<resolver ref="maven2"/>
|
||||||
|
</chain>
|
||||||
|
<chain name="internal">
|
||||||
|
<resolver ref="local"/>
|
||||||
|
</chain>
|
||||||
|
<chain name="external">
|
||||||
|
<resolver ref="maven2"/>
|
||||||
|
</chain>
|
||||||
|
<chain name="external-and-snapshots">
|
||||||
|
<resolver ref="maven2"/>
|
||||||
|
<resolver ref="apache-snapshot"/>
|
||||||
|
</chain>
|
||||||
|
</resolvers>
|
||||||
|
<modules>
|
||||||
|
<!--
|
||||||
|
This forces a requirement for other hadoop-artifacts to be built locally
|
||||||
|
rather than look for them online.
|
||||||
|
|
||||||
|
-->
|
||||||
|
<module organisation="org.apache.hadoop" name=".*" resolver="internal"/>
|
||||||
|
<!--until commons cli is external, we need to pull it in from the snapshot repository -if present -->
|
||||||
|
<module organisation="org.apache.commons" name=".*" resolver="external-and-snapshots"/>
|
||||||
|
</modules>
|
||||||
|
</ivysettings>
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
#This properties file lists the versions of the various artifacts used by hadoop and components.
|
||||||
|
#It drives ivy and the generation of a maven POM
|
||||||
|
|
||||||
|
#These are the versions of our dependencies (in alphabetical order)
|
||||||
|
apacheant.version=1.7.0
|
||||||
|
|
||||||
|
checkstyle.version=4.2
|
||||||
|
|
||||||
|
commons-cli.version=2.0-SNAPSHOT
|
||||||
|
commons-codec.version=1.3
|
||||||
|
commons-collections.version=3.1
|
||||||
|
commons-httpclient.version=3.0.1
|
||||||
|
commons-lang.version=2.4
|
||||||
|
commons-logging.version=1.0.4
|
||||||
|
commons-logging-api.version=1.0.4
|
||||||
|
commons-el.version=1.0
|
||||||
|
commons-fileupload.version=1.2
|
||||||
|
commons-io.version=1.4
|
||||||
|
commons-net.version=1.4.1
|
||||||
|
core.version=3.1.1
|
||||||
|
coreplugin.version=1.3.2
|
||||||
|
|
||||||
|
hsqldb.version=1.8.0.10
|
||||||
|
|
||||||
|
#ivy.version=2.0.0-beta2
|
||||||
|
ivy.version=2.0.0-rc2
|
||||||
|
|
||||||
|
jasper.version=5.5.12
|
||||||
|
#not able to figureout the version of jsp & jsp-api version to get it resolved throught ivy
|
||||||
|
# but still declared here as we are going to have a local copy from the lib folder
|
||||||
|
jsp.version=2.1
|
||||||
|
jsp-api.version=5.5.12
|
||||||
|
jets3t.version=0.6.1
|
||||||
|
jetty.version=6.1.14
|
||||||
|
jetty-util.version=6.1.14
|
||||||
|
junit.version=4.5
|
||||||
|
jdiff.version=1.0.9
|
||||||
|
json.version=1.0
|
||||||
|
|
||||||
|
kfs.version=0.1
|
||||||
|
|
||||||
|
log4j.version=1.2.15
|
||||||
|
lucene-core.version=2.3.1
|
||||||
|
|
||||||
|
oro.version=2.0.8
|
||||||
|
|
||||||
|
rats-lib.version=0.5.1
|
||||||
|
|
||||||
|
servlet.version=4.0.6
|
||||||
|
servlet-api-2.5.version=6.1.14
|
||||||
|
servlet-api.version=2.5
|
||||||
|
slf4j-api.version=1.4.3
|
||||||
|
slf4j-log4j12.version=1.4.3
|
||||||
|
|
||||||
|
xmlenc.version=0.52
|
||||||
|
xerces.version=1.4.4
|
||||||
Loading…
Reference in New Issue