Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,530 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

15530 Commits

Author SHA1 Message Date
Masatake Iwasaki 965fd38000 updated site index for 2.10.2 release. 2022-05-24 19:21:50 +09:00
Owen O'Malley 227d64ab59 YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 
(cherry picked from commit f390edaec4)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java

(cherry picked from commit 88a8752fa2)
 2022-05-24 14:40:57 +09:00
Szilard Nemeth ba041fe6d3 YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 
(cherry picked from commit 45801fba8b)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java

(cherry picked from commit b2be007db5)
 2022-05-24 14:40:40 +09:00
Akira Ajisaka 759c7a0557 HADOOP-15261. Upgrade commons-io from 2.4 to 2.5. Contributed by PandaMonkey. 
(cherry picked from commit 432cd74c72)
(cherry picked from commit 4803fdc476)
 2022-05-24 14:40:28 +09:00
Masatake Iwasaki 38fd628762 Preparing for 2.10.2 release 2022-05-17 00:48:39 +09:00
Ashutosh Gupta ec5f3e4ed1
YARN-11125. Backport YARN-6483 to branch-2.10 (#4258) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:21:37 +09:00
Ashutosh Gupta b4d9905fd6 MAPREDUCE-7246. In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, updating the datatype of appId to "string". (#4223) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fb13c1e4a8)
 2022-04-25 14:35:08 +09:00
Renukaprasad C 8b9b12d1c5
HDFS-16551. CryptoInputStream#close() should be syncronized (#4227) 
(cherry picked from commit 5f259ef7fd)
 2022-04-24 11:25:51 -07:00
Xing Lin e71d57edc7
HADOOP-15783. [JDK10] TestSFTPFileSystem.testGetModifyTime fails. (#4218) 
(cherry picked from commit 93b0f540ed)

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-22 15:42:36 +09:00
Xing Lin ab1cf3cca4
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#4208) 
(cherry picked from commit 328eae9a14)

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Co-authored-by: Xing Lin <xinglin@linkedin.com>
 2022-04-21 07:28:48 -07:00
Xing Lin 96ee4879fc
HADOOP-18172: Change scope of InodeTree and its member methods to make them accessible from outside package. (#4144) 
Reviewed-by: Virajith Jalaparti <virajith@apache.org>
 2022-04-20 16:26:48 -07:00
Xing Lin 4f7b2de97a
Hadoop-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4128) 
* HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS

(cherry picked from commit 15a5ea2c95)

* Removed all other test cases, as InternalDirOfViewFs implementation
in ViewFs is out of date in 2.10 branch.

Co-authored-by: Xing Lin <xinglin@linkedin.com>
 2022-04-15 13:58:14 -07:00
Tom ba4216c677
HADOOP-15133. [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9. (#4173) 
(cherry picked from commit d2d8f4aeb3)

Contributed by Akira Ajisaka
 2022-04-15 13:24:14 +01:00
Wei-Chiu Chuang 10f3f0cf42 HADOOP-18155. Refactor tests in TestFileUtil (#4063) 
(cherry picked from commit d0fa9b5775)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Co-authored-by: Gautham B A <gautham.bangalore@gmail.com>
(cherry picked from commit 743db6e7b4)
(cherry picked from commit 826d13a59a)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
 2022-04-15 07:56:50 +09:00
belugabehr 12c3b8f8df HADOOP-16811: Use JUnit TemporaryFolder Rule in TestFileUtils (#1811). Contributed by David Mollitor. 
(cherry picked from commit 1afd54fbbb)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
 2022-04-15 07:18:31 +09:00
Inigo Goiri 20112bd8e8 HADOOP-15513. Add additional test cases to cover some corner cases for FileUtil#symlink. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit db81f85e84)
 2022-04-14 13:13:42 +09:00
Wei-Chiu Chuang 9a687f4e4b HADOOP-17718. Explicitly set locale in the Dockerfile. (#3034) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ad923ad564)

Conflicts:
	dev-support/docker/Dockerfile_centos_7
	dev-support/docker/Dockerfile_centos_8
(cherry picked from commit 5e09d38b25)

Conflicts:
	dev-support/docker/Dockerfile_aarch64
(cherry picked from commit c531613cc9)

Conflicts:
	dev-support/docker/Dockerfile
 2022-04-14 01:59:14 +00:00
Masatake Iwasaki 6715746806
HADOOP-18088. Replace log4j 1.x with reload4j. (#4151) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-14 09:33:24 +09:00
Viraj Jasani 327a30dc45 HADOOP-18191. Log retry count while handling exceptions in RetryInvocationHandler (#4133) 
(cherry picked from commit b69ede7154)
 2022-04-11 15:24:51 +09:00
Hanisha Koneru 8d24aba731 HADOOP-17116. Skip Retry INFO logging on first failover from a proxy 
(cherry picked from commit e62d8f8412)
 2022-04-11 15:20:43 +09:00
wangzhaohui 0fdcfe2beb HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a7)
 2022-04-06 17:32:22 +09:00
Xing Lin 4bd2f58a83
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS (#4124) 
* HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS.

To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
(cherry picked from commit 8b8158f02d)
 2022-03-31 20:58:29 +00:00
Lei Yang ebaaf3c6dd HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:31:50 -07:00
Akira Ajisaka 37a7b9f934
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. (#4103) 
(cherry picked from commit a0deda1a77)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java

(cherry picked from commit dbeb41b46a)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java

Co-authored-by: Peter Bacsko <pbacsko@cloudera.com>
 2022-03-24 17:37:09 +09:00
Owen O'Malley ce208b1a1e HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-23 15:06:22 -07:00
Owen O'Malley f1c3018d0f
HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Backport of HADOOP-16161. 
Fixes #4091

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-23 14:50:18 -07:00
Abhishek Das a7f4ca55ff
HADOOP-18129: Change URI to String in INodeLink to reduce memory footprint of ViewFileSystem 
Fixes #3996
Fixes #4083

(cherry picked from commit da9970dd69)
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-22 13:54:33 -07:00
Manoj Govindassamy ef8582bfa8 HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem 
(cherry picked from commit 133d7ca76e)
Fixes #4015
 2022-03-15 17:04:50 -07:00
Chris Douglas 20ff294714 HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov 
(cherry picked from commit 1f3bc63e67)
 2022-03-15 15:40:17 -07:00
Andrew Wang e831f7b228 HADOOP-13722. Code cleanup -- ViewFileSystem and InodeTree. Contributed by Manoj Govindassamy. 
(cherry picked from commit 0f4afc8100)
 2022-03-15 15:40:17 -07:00
Masatake Iwasaki efe515dbd5
HADOOP-18158. Fix failure of create-release script due to releasedocmaker changes in branch-2.10 (#4055) 2022-03-10 00:35:47 +09:00
Wanqiang Ji cb91880458 HADOOP-16334. Fix yetus-wrapper not working when HADOOP_YETUS_VERSION >= 0.9.0 
This closes #864

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d23e8e9232)
 2022-03-09 13:05:30 +09:00
Ayush Saxena f36d61a0c2
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
(cherry picked from commit fa30224e95)
(cherry picked from commit 551f8d3ddc)
 2022-02-23 11:42:01 +08:00
Viraj Jasani 2d2fded2e7
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation (#3991) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 697e5d4636)
(cherry picked from commit d763c99707)
(cherry picked from commit 9d618bcc6b)
 2022-02-22 11:02:23 +08:00
Steve Loughran 4313bec40a
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I5bca6bd3e7baa6384f634315c32cf5cdd2efc370
 2022-02-21 18:00:37 +00:00
Xing Lin d027e79bf9
HADOOP-18110. ViewFileSystem: Add Support for Localized Trash Root 
Fixes #3956
Fixes #3994

(cherry picked from commit ca8ba24051)
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-02-16 13:23:46 -08:00
Xiao Chen cb5af0012e HDFS-10650. DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission. Contributed by John Zhuge. 2022-02-15 16:25:25 -08:00
Ashutosh Gupta 819717bf11
HADOOP-18099. Upgrade bundled Tomcat to 8.5.75 (#3947) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-01 14:27:27 +09:00
Xing Lin a63055eda6
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java (#3918). Contributed by Xing Lin. (#3933) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0d17b629ff)
 2022-01-26 21:54:42 +05:30
Ashutosh Gupta b567dcd05f
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a86)
 2022-01-11 02:16:08 +09:00
Ayush Saxena d10bbaf0ef
HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 08:06:27 +05:30
Akira Ajisaka 112efdc74e HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de2550)

 Conflicts:
	dev-support/Jenkinsfile
 2021-12-28 16:13:35 +09:00
Akira Ajisaka aa678699ad HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d)
 2021-12-28 16:10:03 +09:00
Akira Ajisaka 0231191cc8
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (#3832) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/bin/yetus-wrapper
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2021-12-28 16:05:55 +09:00
Akira Ajisaka 87726b3dcc
HADOOP-16054. Update Dockerfile to use Bionic (#3833) 
(cherry picked from commit 81d8b71534)

 Conflicts:
	dev-support/docker/Dockerfile
 2021-12-28 15:25:01 +09:00
Ashutosh Gupta 205dddbfc6 YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
 2021-12-23 17:49:52 +09:00
Eric Payne 9ee5265fb3 YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:48:06 +00:00
Takanobu Asanuma cc842231df HADOOP-15293. TestLogLevel fails on Java 9 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 99ab511cba)

 Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogLevel.java
 2021-12-18 01:24:52 +09:00
Dhananjay Badaya ebf569793b
HADOOP-13500. Synchronizing iteration of Configuration properties object (#3776) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-18 01:07:38 +09:00
Dhananjay Badaya 2873606fc9
HADOOP-18049. Pin python-lazy-object-proxy to 1.6.0 in docker image (#3813) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-17 18:02:26 +09:00