Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,473 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24473 Commits

Author SHA1 Message Date
Ayush Saxena d8314fffc0 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 
(cherry picked from commit c2a17659d1)
 2021-03-22 12:38:09 +00:00
Ayush Saxena 793ab9acf3 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58)
 2021-03-22 12:35:11 +00:00
litao bc2bdee76c HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 
(cherry picked from commit 343ce8a8c9)
 2021-03-22 20:30:53 +09:00
Mingliang Liu 23082ac6c7
YARN-10706. Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 (#2791) 
Contributed by Mingliang Liu.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-21 21:14:12 -07:00
kwangsun 3aae563421
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1)
 2021-03-22 11:45:42 +09:00
Ayush Saxena 0b43c65816 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad)
 2021-03-19 18:50:31 +00:00
Ayush Saxena 9665a0be73 HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:18 +05:30
Ayush Saxena b87c0ea7eb HADOOP-17594. DistCp: Expose the JobId for applications executing through run method (#2786). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-19 14:20:32 +05:30
Ayush Saxena 27944772d3 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:13:31 +05:30
Eric Badger 0e37da334b YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu. 
(cherry picked from commit 181b9ddc51)
 2021-03-18 17:47:18 +00:00
Eric Badger 8c82e3507d YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu. 
(cherry picked from commit 38495af325)
 2021-03-18 17:47:10 +00:00
Eric Badger cd417f17ae YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 
(cherry picked from commit 49f89f1d3d)
 2021-03-17 18:16:59 +00:00
Viraj Jasani 17307ee88d HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348)
 2021-03-17 15:13:27 +09:00
Takanobu Asanuma aec9d2cff7 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8)
 2021-03-17 14:15:53 +09:00
touchida f4a9327002
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7)
 2021-03-17 12:01:15 +09:00
Jim Brennan ad74038e02 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 299b8062f1)
 2021-03-15 20:17:02 +00:00
Eric Badger 1022ed91ac YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.

(cherry picked from commit 7d3c8ef606)
 2021-03-15 20:01:24 +00:00
Eric Payne f5810ea83c YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7)
 2021-03-15 19:14:19 +00:00
Chao Sun e2aaa10512 HADOOP-17532. Yarn Job execution get failed when LZ4 Compression Codec is used. Contributed Bhavik Patel. 2021-03-14 21:20:44 -07:00
He Xiaoqiao 7fb49a48d1 HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0)
 2021-03-14 14:56:16 +00:00
Ayush Saxena 28e89509a3 HADOOP-17586. Upgrade org.codehaus.woodstox:stax2-api to 4.2.1. (#2769). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-03-13 15:10:27 +05:30
Jim Brennan 2204bafbfb [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f3)
 2021-03-12 17:20:37 +00:00
Akira Ajisaka d2723b21b9
HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d)
 2021-03-12 18:05:00 +09:00
Viraj Jasani 36313b38cb
HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns (#2757) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-11 11:55:09 -08:00
Akira Ajisaka 0c1ba49c37
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de2550)
 2021-03-11 23:26:36 +09:00
Steve Loughran 469fcdaf8f HADOOP-16721. Improve S3A rename resilience (#2742) 
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.

When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.

Contributed by Steve Loughran.
 2021-03-11 12:54:15 +00:00
Akira Ajisaka de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-11 14:57:03 +09:00
Hui Fei 34f7562abe HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee4)
 2021-03-11 10:07:48 +08:00
sumangala-patki cdaa64458d
HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results (#2597) 
Contributed by Bilahari T H and Sumangala Patki
 2021-03-10 18:25:41 +00:00
Pierrick Hymbert 0e82d42d2c [HADOOP-17567] typo in MagicCommitTracker (#2749) 
Contributed by Pierrick Hymbert
 2021-03-10 15:41:21 +00:00
Mike 5ffcee8979
HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem (#2701) 
Contributed by Mike Pryakhin.

Change-Id: I59ef67c38c313f30c5e000b2fe41fcf715cf3a4b
 2021-03-09 19:58:11 +00:00
Brahma Reddy Battula f12293fba2 YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit b2a565629d)
 2021-03-09 20:27:07 +05:30
Akira Ajisaka 2da6a96821
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b77)
 2021-03-09 11:10:53 +09:00
Peter Bacsko eb05d49349 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 12:36:08 +01:00
Peter Bacsko 066f89af01 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 11:42:59 +01:00
tomscut c498ef4b6d HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:00:25 -08:00
Eric Badger 1a33878279 YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.

(cherry picked from commit 9d088639bf)
 2021-03-05 19:49:49 +00:00
Takanobu Asanuma e607d03995 HADOOP-17563. Update Bouncy Castle to 1.68. (#2740) 
(cherry picked from commit 0774116756)
 2021-03-05 22:59:09 +09:00
Neil 0396a721e3 YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 
(cherry picked from commit d615e2d3bd)
 2021-03-04 14:54:28 +05:30
Akira Ajisaka 4462da0a84
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa)
 2021-03-04 14:56:54 +09:00
litao 852adff194 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380)
 2021-03-03 22:35:45 +09:00
Eric Badger 138021ce23 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.

(cherry picked from commit a17dc4e821)
 2021-03-02 21:49:09 +00:00
Konstantin V Shvachko 56679e83bf HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:20:40 -08:00
S O'Donnell 24a0304159 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)
 2021-03-01 11:50:23 +00:00
Ahmed Hussein 792329fde9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a06)
 2021-02-26 19:56:07 +00:00
Eric Badger e4dcc31114 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec)
 2021-02-26 19:07:59 +00:00
Surendra Singh Lilhore a6f86af39f HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3)
 2021-02-26 10:49:16 +08:00
Jonathan Hung be6e99963d YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
 2021-02-25 15:09:33 -08:00
Jim Brennan db457b056a [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit c373da9f88)
 2021-02-25 16:48:46 +00:00
S O'Donnell e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:04:52 +00:00