Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,511 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

15511 Commits

Author SHA1 Message Date
Hanisha Koneru 8d24aba731 HADOOP-17116. Skip Retry INFO logging on first failover from a proxy 
(cherry picked from commit e62d8f8412)
 2022-04-11 15:20:43 +09:00
wangzhaohui 0fdcfe2beb HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a7)
 2022-04-06 17:32:22 +09:00
Xing Lin 4bd2f58a83
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS (#4124) 
* HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS.

To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
(cherry picked from commit 8b8158f02d)
 2022-03-31 20:58:29 +00:00
Lei Yang ebaaf3c6dd HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:31:50 -07:00
Akira Ajisaka 37a7b9f934
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. (#4103) 
(cherry picked from commit a0deda1a77)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java

(cherry picked from commit dbeb41b46a)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java

Co-authored-by: Peter Bacsko <pbacsko@cloudera.com>
 2022-03-24 17:37:09 +09:00
Owen O'Malley ce208b1a1e HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-23 15:06:22 -07:00
Owen O'Malley f1c3018d0f
HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Backport of HADOOP-16161. 
Fixes #4091

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-23 14:50:18 -07:00
Abhishek Das a7f4ca55ff
HADOOP-18129: Change URI to String in INodeLink to reduce memory footprint of ViewFileSystem 
Fixes #3996
Fixes #4083

(cherry picked from commit da9970dd69)
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-22 13:54:33 -07:00
Manoj Govindassamy ef8582bfa8 HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem 
(cherry picked from commit 133d7ca76e)
Fixes #4015
 2022-03-15 17:04:50 -07:00
Chris Douglas 20ff294714 HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov 
(cherry picked from commit 1f3bc63e67)
 2022-03-15 15:40:17 -07:00
Andrew Wang e831f7b228 HADOOP-13722. Code cleanup -- ViewFileSystem and InodeTree. Contributed by Manoj Govindassamy. 
(cherry picked from commit 0f4afc8100)
 2022-03-15 15:40:17 -07:00
Masatake Iwasaki efe515dbd5
HADOOP-18158. Fix failure of create-release script due to releasedocmaker changes in branch-2.10 (#4055) 2022-03-10 00:35:47 +09:00
Wanqiang Ji cb91880458 HADOOP-16334. Fix yetus-wrapper not working when HADOOP_YETUS_VERSION >= 0.9.0 
This closes #864

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d23e8e9232)
 2022-03-09 13:05:30 +09:00
Ayush Saxena f36d61a0c2
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
(cherry picked from commit fa30224e95)
(cherry picked from commit 551f8d3ddc)
 2022-02-23 11:42:01 +08:00
Viraj Jasani 2d2fded2e7
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation (#3991) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 697e5d4636)
(cherry picked from commit d763c99707)
(cherry picked from commit 9d618bcc6b)
 2022-02-22 11:02:23 +08:00
Steve Loughran 4313bec40a
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I5bca6bd3e7baa6384f634315c32cf5cdd2efc370
 2022-02-21 18:00:37 +00:00
Xing Lin d027e79bf9
HADOOP-18110. ViewFileSystem: Add Support for Localized Trash Root 
Fixes #3956
Fixes #3994

(cherry picked from commit ca8ba24051)
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-02-16 13:23:46 -08:00
Xiao Chen cb5af0012e HDFS-10650. DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission. Contributed by John Zhuge. 2022-02-15 16:25:25 -08:00
Ashutosh Gupta 819717bf11
HADOOP-18099. Upgrade bundled Tomcat to 8.5.75 (#3947) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-01 14:27:27 +09:00
Xing Lin a63055eda6
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java (#3918). Contributed by Xing Lin. (#3933) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0d17b629ff)
 2022-01-26 21:54:42 +05:30
Ashutosh Gupta b567dcd05f
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a86)
 2022-01-11 02:16:08 +09:00
Ayush Saxena d10bbaf0ef
HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 08:06:27 +05:30
Akira Ajisaka 112efdc74e HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de2550)

 Conflicts:
	dev-support/Jenkinsfile
 2021-12-28 16:13:35 +09:00
Akira Ajisaka aa678699ad HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d)
 2021-12-28 16:10:03 +09:00
Akira Ajisaka 0231191cc8
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (#3832) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/bin/yetus-wrapper
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2021-12-28 16:05:55 +09:00
Akira Ajisaka 87726b3dcc
HADOOP-16054. Update Dockerfile to use Bionic (#3833) 
(cherry picked from commit 81d8b71534)

 Conflicts:
	dev-support/docker/Dockerfile
 2021-12-28 15:25:01 +09:00
Ashutosh Gupta 205dddbfc6 YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
 2021-12-23 17:49:52 +09:00
Eric Payne 9ee5265fb3 YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:48:06 +00:00
Takanobu Asanuma cc842231df HADOOP-15293. TestLogLevel fails on Java 9 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 99ab511cba)

 Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogLevel.java
 2021-12-18 01:24:52 +09:00
Dhananjay Badaya ebf569793b
HADOOP-13500. Synchronizing iteration of Configuration properties object (#3776) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-18 01:07:38 +09:00
Dhananjay Badaya 2873606fc9
HADOOP-18049. Pin python-lazy-object-proxy to 1.6.0 in docker image (#3813) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-17 18:02:26 +09:00
Akira Ajisaka 7d88a54d40 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml

(cherry picked from commit 94ca965e21)
 2021-12-10 01:47:31 +09:00
Akira Ajisaka eda934025b HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 3854b59bd1)
 2021-12-10 01:27:38 +09:00
Ashutosh Gupta 0b9ae87ee7 YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5a950b8900)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
 2021-12-06 20:22:35 +09:00
Sunil G 29f81c6121 YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 215f2052fc)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSPreemptionThread.java
 2021-11-29 14:35:47 +09:00
Shubham Gupta 484cac36fd YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069)
 2021-11-29 14:21:56 +09:00
Yufei Gu 68ac742d94 HADOOP-15331. Fix a race condition causing parsing error of java.io.BufferedInputStream in class org.apache.hadoop.conf.Configuration. Contributed by Miklos Szegedi. 
(cherry picked from commit 268c29a5f5)
 2021-11-29 14:13:05 +09:00
Eric Yang 0d88d832cd YARN-7266. Fixed deadlock in Timeline Server thread initialization. 
Contributed by Prabhu Joseph

(cherry picked from commit 7b42e0e32a)
 2021-11-29 12:43:05 +09:00
Abhishek Das f9463e5111 HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 
(cherry picked from commit 54a1d78e16)
 2021-11-16 22:28:10 -08:00
litao e648b3e34b HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:27:17 +09:00
Gautham B A 1b50c55a21
HADOOP-17988. Disable JIRA plugin for YETUS on Hadoop (#3623) 2021-11-11 00:14:04 +05:30
litao feb5971f49 HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:24:56 +09:00
litao 899c4802fc HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d8)
 2021-11-10 19:19:31 +09:00
jianghuazhu 64cd82e472 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:46:50 +09:00
Zhendong Bai eb6be306b4
HADOOP-17880. Build Hadoop on Centos 7 (#3535) 2021-11-08 17:45:12 +05:30
jianghuazhu f7ec214568 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:57:35 +09:00
Gautham B A 3dd888023b
HADOOP-17978. Exclude ASF license check for pkg-resolver JSON (#3583) 2021-10-25 16:13:26 -07:00
Ahmed Hussein de120b16ad YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-22 17:02:38 +00:00
Michael Kuchenbecker 79d1818879 HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:54:05 -07:00
Szilard Nemeth 5db1b2d7fa HADOOP-17857. Check real user ACLs in addition to proxied user ACLs. Contributed by Eric Payne 
(cherry picked from commit 5428d36b56)
 2021-10-21 17:40:02 +00:00