Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,830 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21830 Commits

Author SHA1 Message Date
Jinhu Wu 2b6f7bf209 HADOOP-17374. support listObjectV2 (#3587) 
(cherry picked from commit a9c51ea57d)
 2021-11-04 21:44:12 -07:00
jianghuazhu 06c4058a61 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:51:14 +09:00
daimin 29fd36e2f1 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java
 2021-11-03 21:30:48 +00:00
litao b1801e99de HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c624)
 2021-10-29 10:29:32 +09:00
Ananya Singh df9c07fc04
HADOOP-17972. Backport HADOOP-17683 for branch-3.2 (#3573) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-28 11:46:12 +09:00
Ahmed Hussein c36d8a1395 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-22 15:54:13 +00:00
Michael Kuchenbecker 7de539f3ca HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:50:54 -07:00
Szilard Nemeth 7ebc18e8fe HADOOP-17857. Check real user ACLs in addition to proxied user ACLs. Contributed by Eric Payne 
(cherry picked from commit 5428d36b56)
 2021-10-21 17:15:39 +00:00
Nicholas Marion 87869c6c26
HADOOP-17971. Exclude IBM Java security classes from being shaded/relocated. (#3567) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3e1135469e)
 2021-10-20 14:30:29 +09:00
daimin 9f2c89e690 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc5)
 2021-10-18 12:20:43 +01:00
Xing Lin c8f22fe737 HADOOP-16532. Fix TestViewFsTrash to use the correct homeDir. Contributed by Xing Lin. (#3514) 
(cherry picked from commit 97c0f96879)
 2021-10-13 14:56:53 -07:00
Masatake Iwasaki c7b40e43f5 HADOOP-17925. BUILDING.txt should not encourage to activate docs profile on building binary artifacts. (#3509) 
(cherry picked from commit 65d9523aa2)
 2021-10-11 08:58:09 +00:00
langlaile1221 88744b8a04
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c8)
(cherry picked from commit a35a6ec06b)
 2021-10-11 16:22:59 +08:00
Brahma Reddy Battula 675712afa3 HADOOP-17236. Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit eb84793af1)
 2021-10-08 16:58:05 +05:30
Siyao Meng c8db6b1eb2
HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3aaac8a1f6)

 Conflicts:
	LICENSE-binary
 2021-10-08 14:58:38 +09:00
Takanobu Asanuma 7a12baeef9 HADOOP-17955. Bump netty to the latest 4.1.68. (#3528) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit cf4d2d826c)
 2021-10-08 10:39:40 +09:00
Ahmed Hussein e99af5cf81 HADOOP-17930. implement non-guava Precondition checkState (#3522) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit c36f9402dc)
 2021-10-07 11:02:51 +09:00
Ahmed Hussein 4b6316a7fd HADOOP-17929. implement non-guava Precondition checkArgument (#3473) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 0c498f21de)
 2021-10-07 11:02:45 +09:00
Rohith Sharma K S 51d17f3cb3 YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C. 
(cherry picked from commit f63e4e4f2e)

Conflicts:
	hadoop-project/pom.xml
 2021-10-06 20:49:00 +09:00
Viraj Jasani 28d4affea0 HADOOP-17947. Additional element types for VisibleForTesting (ADDENDUM) (#3521) 
(cherry picked from commit 783e4805e7)
 2021-10-06 02:19:44 +09:00
Viraj Jasani ce30f796f6 HADOOP-17947. Provide alternative to Guava VisibleForTesting (#3505) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 5b1d594005)
 2021-10-05 10:02:08 +09:00
AngersZhuuuu 1df3b3de46
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4)
 2021-09-29 17:44:32 +08:00
He Xiaoqiao cfd0265975
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe)
 2021-09-29 17:44:00 +08:00
Takanobu Asanuma 3851ace026 HADOOP-17940. Upgrade Kafka to 2.8.1 (#3488) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2068b0041c)
 2021-09-28 13:36:40 +09:00
Inigo Goiri 86028d975c
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dc)
 2021-09-27 14:52:27 +09:00
Wei-Chiu Chuang 978ec57445 HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf)
(cherry picked from commit bb08de559a)
 2021-09-24 09:15:04 -07:00
Akira Ajisaka 85346db636
HADOOP-17917. Backport HADOOP-15993 to branch-3.2 which address CVE-2014-4611. Contributed by Brahma Reddy Battula. 2021-09-24 14:59:21 +09:00
Eric Badger ce3def0aa7 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-22 20:29:12 +00:00
Rintaro Ikeda 5de18fb483 HADOOP-17926. Maven-eclipse-plugin is no longer needed since Eclipse can import Maven projects by itself. (#3465) 
(cherry picked from commit 962068d2d8)
 2021-09-22 13:09:38 +00:00
He Xiaoqiao 4944dafa29
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao 50b0f0dc42
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao fc5fe67b3f
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa1)
 2021-09-17 22:16:11 +08:00
Inigo Goiri 5da46e197b
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5)
 2021-09-17 22:16:11 +08:00
Ayush Saxena f77965b29e
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab)
 2021-09-17 22:16:11 +08:00
Inigo Goiri d5501c125f
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d)
 2021-09-17 22:16:07 +08:00
Rintaro Ikeda a777264c7f HADOOP-17919. Fix command line example in Hadoop Cluster Setup documentation. (#3453) 
(cherry picked from commit 607c20c612)
 2021-09-17 13:35:54 +00:00
Steve Loughran 5094dede8e
HADOOP-17126. implement non-guava Precondition checkNotNull 
This adds a new class org.apache.hadoop.util.Preconditions which is

* @Private/@Unstable
* Intended to allow us to move off Google Guava
* Is designed to be trivially backportable
  (i.e contains no references to guava classes internally)

Please use this instead of the guava equivalents, where possible.

Contributed by: Ahmed Hussein

Change-Id: Ic392451bcfe7d446184b7c995734bcca8c07286e
 2021-09-17 11:37:29 +01:00
wangzhaohui 928dbac0a6 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14)
 2021-09-15 10:06:27 -07:00
Szilard Nemeth 4066cb0793 YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak 2021-09-14 18:09:35 +02:00
Tamas Domok 478d2f2897
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir (#3410) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:35:07 +02:00
EungsopYoo 4eeb4d32e5
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8b)
(cherry picked from commit 51a4a23e37)
 2021-09-14 16:49:14 +08:00
Ahmed Hussein a3920e89de
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 12:31:49 +08:00
LeonGao 3c03ade285
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 
(cherry picked from commit 9434c1eccc)
 2021-09-14 12:31:44 +08:00
Ahmed Hussein 33c60ba060
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 
(cherry picked from commit 98097b8f19)
 2021-09-14 12:31:34 +08:00
Stephen O'Donnell ad55d5d1d2
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0)
 2021-09-14 12:31:20 +08:00
Stephen O'Donnell 7039433146
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed)
 2021-09-14 12:31:00 +08:00
He Xiaoqiao ca182b6e5f
Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200)" 
This reverts commit 318bc5ff69.
 2021-09-14 12:27:42 +08:00
Ahmed Hussein 318bc5ff69
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200) 
* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8f19)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1eccc)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
 2021-09-11 19:52:16 +08:00
Ahmed Hussein 1944e0d714 HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:28:29 +00:00
Masatake Iwasaki 34d443a2f0 HADOOP-17897. Allow nested blocks in switch case in checkstyle settings. (#3394) 
(cherry picked from commit e183ec8998)
 2021-09-08 14:41:56 +09:00