Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,976 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8180 Commits

Author SHA1 Message Date
zhaoyim 9316ca149f HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59)
 2019-10-25 13:10:56 -07:00
Wei-Chiu Chuang 8cf4bc4bda HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc9)
 2019-10-24 13:13:27 -07:00
Wei-Chiu Chuang 570ffa1cd6 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7e)
 2019-10-24 13:13:19 -07:00
Vinayakumar B e1e3d9c81d HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:15:18 +05:30
Mukul Kumar Singh c9d07a42dc HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad8)
 2019-10-22 13:31:34 -07:00
Wei-Chiu Chuang 31243f0d29 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-22 09:34:11 -07:00
Ayush Saxena 8c74717720 HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:45:19 +05:30
Kihwal Lee 2d1f3611cd HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef773)
 2019-10-21 16:34:17 -05:00
Ayush Saxena 0f64ed89d2 HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:42:21 +05:30
Ayush Saxena c2cd847a15 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:29:16 +05:30
Surendra Singh Lilhore 5c50eb2243 HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d72)
 2019-10-17 22:39:57 +05:30
Surendra Singh Lilhore d1b30e8450 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd873)
 2019-10-14 23:47:42 +05:30
Surendra Singh Lilhore 2ca05ca02a HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ac)
 2019-10-09 23:32:21 +05:30
Chen Liang a12ab911ee HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a)
 2019-10-08 12:03:04 -07:00
Ayush Saxena 57ab854492 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:22:16 +05:30
Surendra Singh Lilhore a0cdebc409 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 382967be51)
 2019-10-08 00:26:27 +05:30
He Xiaoqiao e29ae7db12 HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37)
 2019-10-04 12:07:30 -07:00
He Xiaoqiao 80392e94b6 HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2019-10-04 12:06:55 -07:00
Erik Krogen 6630c9b75d HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e00143)
 2019-10-04 11:15:49 -07:00
Erik Krogen 69b0c513a9 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa)
 2019-10-04 10:48:33 -07:00
Eric Yang 8bb2b00d38 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0)
 2019-10-04 13:16:08 -04:00
Siyao Meng 673c9d53ca HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b)
 2019-10-04 09:40:22 -07:00
Ayush Saxena 702572434c HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8)
 2019-10-04 08:52:59 -07:00
Takanobu Asanuma 732a68cfb4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085)
 2019-10-04 16:48:50 +09:00
Stephen O'Donnell 966193153f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java
 2019-10-03 22:18:29 -07:00
Inigo Goiri 94af96325c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b2274)
 2019-10-03 18:21:46 -07:00
Inigo Goiri 21a89d544f HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff5)
 2019-10-03 17:27:44 -07:00
Shashikant Banerjee f14fb9081f HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a)
 2019-10-03 17:27:06 -07:00
Vinayakumar B 4e223d9968 HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a530)
 2019-10-03 17:15:45 -07:00
Kitti Nanasi cb701f4da0 HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc9200)
 2019-10-03 16:18:23 -07:00
Shweta Yakkali eaf09e51ba HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:13:44 -07:00
Ayush Saxena ebd7a0f155 HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:00:38 -07:00
Shweta Yakkali 3186afa24c HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 15:58:00 -07:00
Ayush Saxena 9a9edfa87e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 
(cherry picked from commit c4c8d5fd0e)
 2019-10-03 14:56:09 -07:00
Chen Liang 05a1eb788f HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-10-02 21:12:54 -07:00
LeonGao91 885dd17595 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 
(cherry picked from commit 9a1d8cfaf5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
 2019-10-02 17:35:33 -07:00
Chen Liang 8a866f01c9 HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:39:26 -07:00
paulward24 0559c1dba3 HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 
(cherry picked from commit d203045c30)
 2019-10-02 15:07:40 -07:00
paulward24 76a9135947 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 
(cherry picked from commit d8bac50e12)
 2019-10-02 14:31:13 -07:00
Inigo Goiri 6148acebc3 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 
(cherry picked from commit 944adc61b1)
 2019-10-02 13:55:47 -07:00
Ayush Saxena 2bb4ac18ec HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 
(cherry picked from commit 66400c1cbb)
 2019-10-02 13:49:12 -07:00
Surendra Singh Lilhore 7d7330aff0 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 
(cherry picked from commit 4852a90e4b)
 2019-10-02 13:39:40 -07:00
Inigo Goiri d2a7a128d9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0c)
 2019-10-01 17:33:14 -07:00
Yiqun Lin c74027d9d3 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee8)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
 2019-10-01 17:14:46 -07:00
Inigo Goiri 3510a9e5e5 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb8)
 2019-10-01 16:57:20 -07:00
Inigo Goiri 917e53f066 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 
(cherry picked from commit 865c328930)
 2019-10-01 09:53:36 -07:00
Inigo Goiri fc8a7a9e5b HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 
(cherry picked from commit 0e79a86582)
 2019-10-01 09:35:04 -07:00
Inigo Goiri 06b2eceb76 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 
(cherry picked from commit be6c8014e6)
 2019-10-01 09:22:15 -07:00
Konstantin V Shvachko 665360ec6f HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 16:57:56 -07:00
Konstantin V Shvachko 3912a6fc7d Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-30 16:57:26 -07:00