Go to file
Steve Loughran 366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
dev-support HADOOP-15925. The config and log of gpg-agent are removed in create-release script. Contributed by Dinesh Chitlangia. 2018-11-16 13:16:59 +09:00
hadoop-assemblies YARN-9016 DocumentStore as a backend for ATSv2. Contributed by Sushil Ks. 2019-03-13 16:45:23 -07:00
hadoop-build-tools HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 2019-04-03 19:35:02 +09:00
hadoop-client-modules HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
hadoop-cloud-storage-project Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
hadoop-common-project HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 2019-04-03 21:23:40 +01:00
hadoop-dist Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
hadoop-hdds HDDS-1365. Fix error handling in KeyValueContainerCheck. Contributed by Supratim Deka. 2019-04-03 14:01:30 +08:00
hadoop-hdfs-project HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
hadoop-mapreduce-project HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
hadoop-maven-plugins HADOOP-15935. [JDK 11] Update maven.plugin-tools.version to 3.6.0. Contributed by Dinesh Chitlangia. 2018-12-20 14:17:32 +09:00
hadoop-minicluster YARN-8899. Fixed minicluster dependency on yarn-server-web-proxy. 2018-10-18 15:46:15 -04:00
hadoop-ozone HDDS-1330 : Add a docker compose for Ozone deployment with Recon. (#669) 2019-04-03 13:20:51 -07:00
hadoop-project HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
hadoop-project-dist Make 3.2.0 aware to other branches - jdiff 2019-01-21 22:41:26 +05:30
hadoop-submarine SUBMARINE-1. Move code base of submarine from yarn-applications to top directory. Contributed by Wangda Tan. 2019-02-17 20:03:56 +05:30
hadoop-tools HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 2019-04-03 21:23:40 +01:00
hadoop-yarn-project HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
.gitattributes HADOOP-13598. Add eol=lf for unix format files in .gitattributes. Contributed by Yiqun Lin. 2016-09-14 11:14:31 +09:00
.gitignore HDDS-1115. Provide ozone specific top-level pom.xml. 2019-02-24 14:40:52 -08:00
BUILDING.txt HDFS-14213. Remove Jansson from BUILDING.txt. Contributed by Dinesh Chitlangia. 2019-01-18 15:31:45 +09:00
Jenkinsfile HADOOP-16035. Jenkinsfile for Hadoop 2019-02-21 07:36:15 -08:00
LICENSE.txt HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
NOTICE.txt YARN-8914. Add xtermjs to YARN UI2. Contributed by Eric Yang and Akhil PB 2018-12-07 10:56:17 -08:00
README.txt HADOOP-15832. Addendum: Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter 2018-10-12 09:40:34 -07:00
pom.ozone.xml HDDS-1115. Provide ozone specific top-level pom.xml. 2019-02-24 14:40:52 -08:00
pom.xml SUBMARINE-41. Fix ASF warnings related to submarine target files. Contributed by Dinesh Chitlangia. This closes #559. 2019-03-07 10:36:32 +05:30
start-build-env.sh HADOOP-14739. Update start-build-env.sh and build instruction for docker for Mac instead of docker toolbox. Contributed by Dinesh Chitlangia. 2018-11-20 23:59:49 +09:00

README.txt

For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/

and our wiki, at:

   http://wiki.apache.org/hadoop/

This distribution includes cryptographic software.  The country in 
which you currently reside may have restrictions on the import, 
possession, use, and/or re-export to another country, of 
encryption software.  BEFORE using any encryption software, please 
check your country's laws, regulations and policies concerning the
import, possession, or use, and re-export of encryption software, to 
see if this is permitted.  See <http://www.wassenaar.org/> for more
information.

The U.S. Government Department of Commerce, Bureau of Industry and
Security (BIS), has classified this software as Export Commodity 
Control Number (ECCN) 5D002.C.1, which includes information security
software using or performing cryptographic functions with asymmetric
algorithms.  The form and manner of this Apache Software Foundation
distribution makes it eligible for export under the License Exception
ENC Technology Software Unrestricted (TSU) exception (see the BIS 
Export Administration Regulations, Section 740.13) for both object 
code and source code.

The following provides more details on the included cryptographic
software:
  Hadoop Core uses the SSL libraries from the Jetty project written 
by mortbay.org.
  Hadoop Yarn Server Web Proxy uses the BouncyCastle Java
cryptography APIs written by the Legion of the Bouncy Castle Inc.