Go to file
Steve Loughran 3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:42:08 +01:00
dev-support HADOOP-15925. The config and log of gpg-agent are removed in create-release script. Contributed by Dinesh Chitlangia. 2018-11-16 13:18:17 +09:00
hadoop-assemblies Preparing for 3.1.3 development 2019-01-07 10:04:58 -08:00
hadoop-build-tools HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 2019-04-03 19:40:23 +09:00
hadoop-client-modules Preparing for 3.1.3 development 2019-01-07 10:04:58 -08:00
hadoop-cloud-storage-project Preparing for 3.1.3 development 2019-01-07 10:04:58 -08:00
hadoop-common-project HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 2019-04-03 21:42:08 +01:00
hadoop-dist Preparing for 3.1.3 development 2019-01-07 10:04:58 -08:00
hadoop-hdfs-project HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 2019-04-03 12:41:07 -07:00
hadoop-mapreduce-project MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal. 2019-04-01 10:39:18 -07:00
hadoop-maven-plugins Preparing for 3.1.3 development 2019-01-07 10:04:58 -08:00
hadoop-minicluster Preparing for 3.1.3 development 2019-01-07 10:04:58 -08:00
hadoop-project HADOOP-16089. AliyunOSS: update oss-sdk version to 3.4.1. Contributed by wujinhu. 2019-02-04 14:48:04 +09:00
hadoop-project-dist Make upstream aware of 3.1.2 release 2019-02-05 14:09:33 -08:00
hadoop-tools HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 2019-04-03 21:42:08 +01:00
hadoop-yarn-project YARN-9391. Fixed node manager environment leaks into Docker containers. 2019-03-25 15:55:46 -04:00
.gitattributes HADOOP-13598. Add eol=lf for unix format files in .gitattributes. Contributed by Yiqun Lin. 2016-09-14 11:14:31 +09:00
.gitignore YARN-7492. Set up SASS for new YARN UI styling. Contributed by Vasudevan Skm. 2017-11-16 13:44:21 +05:30
BUILDING.txt HDFS-14213. Remove Jansson from BUILDING.txt. Contributed by Dinesh Chitlangia. 2019-01-18 15:34:07 +09:00
LICENSE.txt HADOOP-15900. Update JSch versions in LICENSE.txt. 2018-11-05 17:52:21 +09:00
NOTICE.txt HADOOP-15899. Update AWS Java SDK versions in NOTICE.txt. 2018-11-05 15:31:53 +09:00
README.txt HADOOP-10956. Fix create-release script to include docs and necessary txt files. (kasha) 2014-09-03 09:39:02 -07:00
pom.xml Preparing for 3.1.3 development 2019-01-07 10:04:58 -08:00
start-build-env.sh HADOOP-15912. start-build-env.sh still creates an invalid /etc/sudoers.d/hadoop-build-${USER_ID} file entry after HADOOP-15802. Contributed by Akira Ajisaka. 2018-11-13 13:59:09 +09:00

README.txt

For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/core/

and our wiki, at:

   http://wiki.apache.org/hadoop/

This distribution includes cryptographic software.  The country in 
which you currently reside may have restrictions on the import, 
possession, use, and/or re-export to another country, of 
encryption software.  BEFORE using any encryption software, please 
check your country's laws, regulations and policies concerning the
import, possession, or use, and re-export of encryption software, to 
see if this is permitted.  See <http://www.wassenaar.org/> for more
information.

The U.S. Government Department of Commerce, Bureau of Industry and
Security (BIS), has classified this software as Export Commodity 
Control Number (ECCN) 5D002.C.1, which includes information security
software using or performing cryptographic functions with asymmetric
algorithms.  The form and manner of this Apache Software Foundation
distribution makes it eligible for export under the License Exception
ENC Technology Software Unrestricted (TSU) exception (see the BIS 
Export Administration Regulations, Section 740.13) for both object 
code and source code.

The following provides more details on the included cryptographic
software:
  Hadoop Core uses the SSL libraries from the Jetty project written 
by mortbay.org.