HADOOP-12651. Replace dev-support with wrappers to Yetus (aw)

This commit is contained in:
Allen Wittenauer 2016-01-21 13:03:34 -08:00
parent 8cecad2d56
commit 2a867355df
6 changed files with 304 additions and 0 deletions

57
dev-support/README.md Normal file
View File

@ -0,0 +1,57 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
This directory contains tools to help in the development and release of Apache Hadoop.
* bin
* releasedocmaker
Build release notes for a given Hadoop project or subproject. This is called from Maven when -Preleasedocs is used. See BUILDING.txt for more information.
* shelldocs
Create documentation for the Unix Shell API. This is called from Maven when -Pshelldocs is used.
* smart-apply-patch
Intelligently apply a patch file to a source tree.
* test-patch
Test a patch against a source tree.
* create-release.sh
Helps REs create a release of Apache Hadoop for distribution.
* determine-flaky-tests-hadoop.py
Given a jenkins test job, this script examines all runs of the job done within specified period of time (number of days prior to the execution time of this script), and reports all failed tests.
* docker
Various helpers for the start-build-env.sh script, including the Dockerfile itself. See parent BUILDING.txt for more information.
* findHangingTest.sh
Finds hanging test from Jenkins build output.
Previously, the scripts test-patch.sh, smart-apply-patch.sh, releasedocmaker.py, and shelldocs.py were in this directory. They have been moved to the Apache Yetus project (https://yetus.apache.org). These scripts have been replaced with wrapper scripts located in the bin directory. Command line options are generally different than the previous versions that shipped with older versions of Apache Hadoop.
The wrapper scripts will download, verify (if GPG is installed), and cache a local copy of Apache Yetus in the hadoop/patchprocess directory. The version that is used may be overridden by setting the HADOOP\_YETUS\_VERSION environment variable. The cache directory may be overwritten by setting the HADOOP\_PATCHPROCESS directory. If a local version of Apache Yetus is already installed, it may be used instead by setting the YETUS\_HOME environment variable to point to that directory.

18
dev-support/bin/releasedocmaker Executable file
View File

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
exec "${BINDIR}/yetus-wrapper" releasedocmaker "$@"

18
dev-support/bin/shelldocs Executable file
View File

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
exec "${BINDIR}/yetus-wrapper" shelldocs "$@"

View File

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
exec "${BINDIR}/yetus-wrapper" smart-apply-patch --project=hadoop "$@"

18
dev-support/bin/test-patch Executable file
View File

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop --skip-dir=dev-support "$@"

175
dev-support/bin/yetus-wrapper Executable file
View File

@ -0,0 +1,175 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o pipefail
## @description Print a message to stderr
## @audience public
## @stability stable
## @replaceable no
## @param string
function yetus_error
{
echo "$*" 1>&2
}
## @description Given a filename or dir, return the absolute version of it
## @audience public
## @stability stable
## @param directory
## @replaceable no
## @return 0 success
## @return 1 failure
## @return stdout abspath
function yetus_abs
{
declare obj=$1
declare dir
declare fn
if [[ ! -e ${obj} ]]; then
return 1
elif [[ -d ${obj} ]]; then
dir=${obj}
else
dir=$(dirname -- "${obj}")
fn=$(basename -- "${obj}")
fn="/${fn}"
fi
dir=$(cd -P -- "${dir}" >/dev/null 2>/dev/null && pwd -P)
if [[ $? = 0 ]]; then
echo "${dir}${fn}"
return 0
fi
return 1
}
WANTED="$1"
shift
ARGV=("$@")
HADOOP_YETUS_VERSION=${HADOOP_YETUS_VERSION:-0.1.0}
BIN=$(yetus_abs "${BASH_SOURCE-$0}")
BINDIR=$(dirname "${BIN}")
###
### if YETUS_HOME is set, then try to use it
###
if [[ -n "${YETUS_HOME}"
&& -x "${YETUS_HOME}/bin/${WANTED}" ]]; then
exec "${YETUS_HOME}/bin/${WANTED}" "${ARGV[@]}"
fi
#
# this directory is ignored by git and maven
#
HADOOP_PATCHPROCESS=${HADOOP_PATCHPROCESS:-"${BINDIR}/../../patchprocess"}
if [[ ! -d "${HADOOP_PATCHPROCESS}" ]]; then
mkdir -p "${HADOOP_PATCHPROCESS}"
fi
mytmpdir=$(yetus_abs "${HADOOP_PATCHPROCESS}")
if [[ $? != 0 ]]; then
yetus_error "yetus-dl: Unable to cwd to ${HADOOP_PATCHPROCESS}"
exit 1
fi
HADOOP_PATCHPROCESS=${mytmpdir}
##
## if we've already DL'd it, then short cut
##
if [[ -x "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then
exec "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}"
fi
##
## need to DL, etc
##
BASEURL="https://dist.apache.org/repos/dist/release/yetus/${HADOOP_YETUS_VERSION}/"
TARBALL="yetus-${HADOOP_YETUS_VERSION}-bin.tar"
GPGBIN=$(command -v gpg)
CURLBIN=$(command -v curl)
pushd "${HADOOP_PATCHPROCESS}" >/dev/null
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}"
exit 1
fi
if [[ -n "${CURLBIN}" ]]; then
"${CURLBIN}" -s -L -O "${BASEURL}/${TARBALL}.gz"
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: unable to download ${BASEURL}/${TARBALL}.gz"
exit 1
fi
else
yetus_error "ERROR: yetus-dl requires curl."
exit 1
fi
if [[ -n "${GPGBIN}" ]]; then
mkdir -p .gpg
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: Unable to create ${HADOOP_PATCHPROCESS}/.gpg"
exit 1
fi
chmod -R 700 .gpg
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: Unable to chmod ${HADOOP_PATCHPROCESS}/.gpg"
exit 1
fi
"${CURLBIN}" -s -L -o KEYS_YETUS https://dist.apache.org/repos/dist/release/yetus/KEYS
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: unable to fetch https://dist.apache.org/repos/dist/release/yetus/KEYS"
exit 1
fi
"${CURLBIN}" -s -L -O "${BASEURL}/${TARBALL}.gz.asc"
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: unable to fetch ${BASEURL}/${TARBALL}.gz.asc"
exit 1
fi
"${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --import "${HADOOP_PATCHPROCESS}/KEYS_YETUS" >/dev/null 2>&1
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: gpg unable to import ${HADOOP_PATCHPROCESS}/KEYS_YETUS"
exit 1
fi
"${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --verify "${TARBALL}.gz.asc" >/dev/null 2>&1
if [[ $? != 0 ]]; then
yetus_error "ERROR: yetus-dl: gpg verify of tarball in ${HADOOP_PATCHPROCESS} failed"
exit 1
fi
fi
gunzip -c "${TARBALL}.gz" | tar xpf -
if [[ $? != 0 ]]; then
yetus_error "ERROR: ${TARBALL}.gz is corrupt. Investigate and then remove ${HADOOP_PATCHPROCESS} to try again."
exit 1
fi
if [[ -x "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then
exec "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}"
fi
##
## give up
##
yetus_error "ERROR: ${WANTED} is not part of Apache Yetus ${HADOOP_YETUS_VERSION}"
exit 1