Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,055 Commits 326 Branches 371 Tags
Commit Graph

8212 Commits

Author SHA1 Message Date
Erik Krogen
6630c9b75d HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e0014343f60f853cb796781ca1fa03a72efd)
 2019-10-04 11:15:49 -07:00
Erik Krogen
69b0c513a9 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa2ef3cd9dd54a6c5009029721e030ed)
 2019-10-04 10:48:33 -07:00
Eric Yang
8bb2b00d38 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0e9812d1f787efb3c40155b0fdeed9c)
 2019-10-04 13:16:08 -04:00
Siyao Meng
673c9d53ca HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b384af93726e4f2e6f5e28c6e3a8f65)
 2019-10-04 09:40:22 -07:00
Ayush Saxena
702572434c HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46)
 2019-10-04 08:52:59 -07:00
Takanobu Asanuma
732a68cfb4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085dbc561c785cef1613bacaf6735d909)
 2019-10-04 16:48:50 +09:00
Stephen O'Donnell
966193153f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167ff9566012ef32104a3964887d62c899)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java
 2019-10-03 22:18:29 -07:00
Inigo Goiri
94af96325c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b227495d748abe8884eee34db507476cee1)
 2019-10-03 18:21:46 -07:00
Inigo Goiri
21a89d544f HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff54f79de0e6439d6c77c2ed437989f10)
 2019-10-03 17:27:44 -07:00
Shashikant Banerjee
f14fb9081f HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a53757f8081e8224e0d4b557fe6a0e2)
 2019-10-03 17:27:06 -07:00
Vinayakumar B
4e223d9968 HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a5305bea1a40c87a4abc0793802bea5c75)
 2019-10-03 17:15:45 -07:00
Kitti Nanasi
cb701f4da0 HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc92009fec2f0ab341f3d878e1b8864b8ea9)
 2019-10-03 16:18:23 -07:00
Shweta Yakkali
eaf09e51ba HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:13:44 -07:00
Ayush Saxena
ebd7a0f155 HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:00:38 -07:00
Shweta Yakkali
3186afa24c HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 15:58:00 -07:00
Ayush Saxena
9a9edfa87e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 
(cherry picked from commit c4c8d5fd0e3c17ccdcf18ece8e005f510328b060)
 2019-10-03 14:56:09 -07:00
Chen Liang
05a1eb788f HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-10-02 21:12:54 -07:00
LeonGao91
885dd17595 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 
(cherry picked from commit 9a1d8cfaf50ec29ffb2d8522ba2f4bc6605d8b8b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
 2019-10-02 17:35:33 -07:00
Chen Liang
8a866f01c9 HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:39:26 -07:00
paulward24
0559c1dba3 HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 
(cherry picked from commit d203045c3024b134d7a0417d1ea3a60d03a1534a)
 2019-10-02 15:07:40 -07:00
paulward24
76a9135947 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 
(cherry picked from commit d8bac50e12d243ef8fd2c7e0ce5c9997131dee74)
 2019-10-02 14:31:13 -07:00
Inigo Goiri
6148acebc3 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 
(cherry picked from commit 944adc61b1830388d520d4052fc7eb6c7ba2790d)
 2019-10-02 13:55:47 -07:00
Ayush Saxena
2bb4ac18ec HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 
(cherry picked from commit 66400c1cbb2b4b2f08f7db965c8b7237072bdcc4)
 2019-10-02 13:49:12 -07:00
Surendra Singh Lilhore
7d7330aff0 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 
(cherry picked from commit 4852a90e4b077ece2d68595210e62959a9923683)
 2019-10-02 13:39:40 -07:00
Inigo Goiri
d2a7a128d9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0cc7d8528e9b4411457f04a528601c69)
 2019-10-01 17:33:14 -07:00
Yiqun Lin
c74027d9d3 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee871df1111ff77d0f6921b13c8ffb50e)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
 2019-10-01 17:14:46 -07:00
Inigo Goiri
3510a9e5e5 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb89f4d1edbb54360622adc61ffacbde3)
 2019-10-01 16:57:20 -07:00
Inigo Goiri
917e53f066 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 
(cherry picked from commit 865c3289308327788f3bed355864c510deb40956)
 2019-10-01 09:53:36 -07:00
Inigo Goiri
fc8a7a9e5b HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 
(cherry picked from commit 0e79a865822eed05f3f8433976b2cfef8f427f25)
 2019-10-01 09:35:04 -07:00
Inigo Goiri
06b2eceb76 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 
(cherry picked from commit be6c8014e66be919388269b70cb2966c35b8c578)
 2019-10-01 09:22:15 -07:00
Konstantin V Shvachko
665360ec6f HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 16:57:56 -07:00
Konstantin V Shvachko
3912a6fc7d Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f6e96c332743a39f965de7995b67bde.
 2019-09-30 16:57:26 -07:00
Ayush Saxena
b7ee3938e9 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-28 08:29:58 +05:30
Chen Liang
2d7b37ea2b HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:55:10 -07:00
Ayush Saxena
a0db762206 HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:21:14 +05:30
Rohith Sharma K S
f9f0338104 Make upstream aware of 3.2.1 release. 2019-09-23 12:03:53 +05:30
Chen Liang
4299bc0d3e HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 16:48:58 -07:00
Kihwal Lee
e29f91bad3 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 1851d06eb3b70f39f3054a7c06f0ad2bc664aaec)
 2019-09-18 15:53:08 -05:00
Wei-Chiu Chuang
b906f53b0d HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 5dd859a8a0d038ff04acbb96741a215d6a0e8c29)
 2019-09-18 09:36:10 -07:00
Anu Engineer
2e9482bd20 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.

(cherry picked from commit 8aaf5e1a14e577a7d8142bc7d49bb94014032afd)
 2019-09-18 15:23:28 +08:00
Ayush Saxena
4126e67e31 HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-09-14 10:12:17 +05:30
Surendra Singh Lilhore
eb1ddcd04c HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 
(cherry picked from commit d1c303a49763029fffa5164295034af8e81e74a0)
 2019-09-12 19:25:50 +05:30
Takanobu Asanuma
8e27d6b4c9 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 
(cherry picked from commit c255333e20c9af6166db5931d70151011d540359)
 2019-09-11 17:03:23 +09:00
Rohith Sharma K S
7d5bb2ebb7 Preparing for 3.2.2-SNAPSHOT development. 2019-09-07 08:52:08 +05:30
Xiaoyu Yao
ebb3b8add0
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 15:33:54 -07:00
Xiaoyu Yao
53d1bfe680
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit 259c6bb0c7fc814ba463a7bf3a786670dccd4c73.
 2019-09-05 15:31:14 -07:00
Xiaoyu Yao
259c6bb0c7
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 09:12:19 -07:00
Stephen O'Donnell
61fd1a74a1 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 915cbc91c0a12cc7b4d3ef4ea951941defbbcb33)
 2019-09-02 10:06:39 -07:00
Wei-Chiu Chuang
84b2fe4b10 Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit 9c0d6e16573a055665318bce8f7d38a92c897393.
 2019-09-02 10:06:39 -07:00
Ayush Saxena
3edffd90c9 HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:58:12 +05:30