Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,756 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21756 Commits

Author SHA1 Message Date
Viraj Jasani 03c908fb04
HDFS-16171. De-flake testDecommissionStatus (#3280) (#3720) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-25 22:18:01 +09:00
GuoPhilipse 253883fd8d
YARN-11007. Correct words in YARN documents (#3680) 
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c9d64bad37)
 2021-11-25 17:51:58 +09:00
Viraj Jasani 2ff303ea00 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2021-11-25 10:26:57 +08:00
Ayush Saxena 4231067d0b
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.. 2021-11-25 02:01:36 +05:30
Viraj Jasani af452d2fc5
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2021-11-24 22:56:44 +09:00
Istvan Fajth dbcadcc21e HADOOP-17975. Fallback to simple auth does not work for a secondary DistributedFileSystem instance. (#3579) 
(cherry picked from commit ae3ba45db5)
 2021-11-24 11:20:27 +00:00
Ashutosh Gupta 850224255c
YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 08f3df3ea2)
 2021-11-24 18:50:31 +09:00
Mukul Kumar Singh f5ff4e1ca8 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2021-11-23 13:53:22 +00:00
litao 2b0339cd34
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
(cherry picked from commit d746e2cdd8)
 2021-11-23 15:43:36 +08:00
smarthan 6e179d421d HADOOP-17998. Allow get command to run with multi threads. (#3645) 
(cherry picked from commit 63018dc73f)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java

(cherry picked from commit cbb3ba135c)
 2021-11-22 13:39:34 +00:00
litao abf8572a16 HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2021-11-19 11:39:48 +09:00
GuoPhilipse 583a41090c
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2021-11-19 01:02:49 +09:00
Viraj Jasani 12960d7fd4
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2021-11-19 00:56:38 +09:00
litao 381e41b97c
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2021-11-19 00:43:42 +09:00
Shubham Gupta ccb711544b
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069)
 2021-11-19 00:25:14 +09:00
Takanobu Asanuma 210464b08f
HDFS-15963. Unreleased volume references cause an infinite loop. Contributed by Shuyan Zhang. (#2941, #3668) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fd)

Co-authored-by: zhangshuyan0 <81411509+zhangshuyan0@users.noreply.github.com>
 2021-11-18 16:32:54 +09:00
Abhishek Das 0e02f03eae HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 
(cherry picked from commit 54a1d78e16)
 2021-11-16 22:21:14 -08:00
litao 81f6cbc715 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 
(cherry picked from commit 026d5860cb)
 2021-11-17 13:53:33 +09:00
Neil 78c56f39b4
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
(cherry picked from commit ce3a7155335416ed4665f90f0ddd7ac3a3edd7e6)
 2021-11-16 22:59:35 +08:00
Gautham B A 980c35ec2d
HADOOP-17993. Disable JIRA plugin for YETUS on Hadoop (#3624) 2021-11-12 19:56:42 +05:30
Wei-Chiu Chuang 002b5fbdc0
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
(cherry picked from commit 5333e872e2)
 2021-11-12 15:03:15 +08:00
litao c95df2eb6f HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:26:19 +09:00
litao ee871a6afd HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:24:32 +09:00
litao ba61dd9b73 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d8)
 2021-11-10 19:18:56 +09:00
jianghuazhu 9ddafb2eee HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:28 +09:00
litao b103be470c HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bca)
 2021-11-09 17:10:01 +09:00
Jinhu Wu 2b6f7bf209 HADOOP-17374. support listObjectV2 (#3587) 
(cherry picked from commit a9c51ea57d)
 2021-11-04 21:44:12 -07:00
jianghuazhu 06c4058a61 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:51:14 +09:00
daimin 29fd36e2f1 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java
 2021-11-03 21:30:48 +00:00
litao b1801e99de HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c624)
 2021-10-29 10:29:32 +09:00
Ananya Singh df9c07fc04
HADOOP-17972. Backport HADOOP-17683 for branch-3.2 (#3573) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-28 11:46:12 +09:00
Ahmed Hussein c36d8a1395 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-22 15:54:13 +00:00
Michael Kuchenbecker 7de539f3ca HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:50:54 -07:00
Szilard Nemeth 7ebc18e8fe HADOOP-17857. Check real user ACLs in addition to proxied user ACLs. Contributed by Eric Payne 
(cherry picked from commit 5428d36b56)
 2021-10-21 17:15:39 +00:00
Nicholas Marion 87869c6c26
HADOOP-17971. Exclude IBM Java security classes from being shaded/relocated. (#3567) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3e1135469e)
 2021-10-20 14:30:29 +09:00
daimin 9f2c89e690 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc5)
 2021-10-18 12:20:43 +01:00
Xing Lin c8f22fe737 HADOOP-16532. Fix TestViewFsTrash to use the correct homeDir. Contributed by Xing Lin. (#3514) 
(cherry picked from commit 97c0f96879)
 2021-10-13 14:56:53 -07:00
Masatake Iwasaki c7b40e43f5 HADOOP-17925. BUILDING.txt should not encourage to activate docs profile on building binary artifacts. (#3509) 
(cherry picked from commit 65d9523aa2)
 2021-10-11 08:58:09 +00:00
langlaile1221 88744b8a04
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c8)
(cherry picked from commit a35a6ec06b)
 2021-10-11 16:22:59 +08:00
Brahma Reddy Battula 675712afa3 HADOOP-17236. Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit eb84793af1)
 2021-10-08 16:58:05 +05:30
Siyao Meng c8db6b1eb2
HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3aaac8a1f6)

 Conflicts:
	LICENSE-binary
 2021-10-08 14:58:38 +09:00
Takanobu Asanuma 7a12baeef9 HADOOP-17955. Bump netty to the latest 4.1.68. (#3528) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit cf4d2d826c)
 2021-10-08 10:39:40 +09:00
Ahmed Hussein e99af5cf81 HADOOP-17930. implement non-guava Precondition checkState (#3522) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit c36f9402dc)
 2021-10-07 11:02:51 +09:00
Ahmed Hussein 4b6316a7fd HADOOP-17929. implement non-guava Precondition checkArgument (#3473) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 0c498f21de)
 2021-10-07 11:02:45 +09:00
Rohith Sharma K S 51d17f3cb3 YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C. 
(cherry picked from commit f63e4e4f2e)

Conflicts:
	hadoop-project/pom.xml
 2021-10-06 20:49:00 +09:00
Viraj Jasani 28d4affea0 HADOOP-17947. Additional element types for VisibleForTesting (ADDENDUM) (#3521) 
(cherry picked from commit 783e4805e7)
 2021-10-06 02:19:44 +09:00
Viraj Jasani ce30f796f6 HADOOP-17947. Provide alternative to Guava VisibleForTesting (#3505) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 5b1d594005)
 2021-10-05 10:02:08 +09:00
AngersZhuuuu 1df3b3de46
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4)
 2021-09-29 17:44:32 +08:00
He Xiaoqiao cfd0265975
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe)
 2021-09-29 17:44:00 +08:00
Takanobu Asanuma 3851ace026 HADOOP-17940. Upgrade Kafka to 2.8.1 (#3488) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2068b0041c)
 2021-09-28 13:36:40 +09:00