Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,708 Commits 326 Branches 371 Tags
Commit Graph

7581 Commits

Author SHA1 Message Date
paulward24
ab7ecd6a0f HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 
(cherry picked from commit d203045c3024b134d7a0417d1ea3a60d03a1534a)
 2019-10-02 15:07:34 -07:00
paulward24
d95dd33ea1 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 
(cherry picked from commit d8bac50e12d243ef8fd2c7e0ce5c9997131dee74)
(cherry picked from commit 76a91359475444cb6d415e5056430c0e743b88ee)
 2019-10-02 14:31:23 -07:00
Ayush Saxena
122b02e288 HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 
(cherry picked from commit 66400c1cbb2b4b2f08f7db965c8b7237072bdcc4)
(cherry picked from commit 2bb4ac18ec0ea8804b424b593615bb06e21a4b6c)
 2019-10-02 13:49:20 -07:00
Surendra Singh Lilhore
f0bdab02e7 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 
(cherry picked from commit d1c303a49763029fffa5164295034af8e81e74a0)
(cherry picked from commit eb1ddcd04c9b0457e19fcc3b320d5b86cc1fda64)
 2019-10-02 13:41:23 -07:00
Inigo Goiri
f1fe3abac9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0cc7d8528e9b4411457f04a528601c69)
 2019-10-01 17:33:08 -07:00
Yiqun Lin
3aea2bcf12 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee871df1111ff77d0f6921b13c8ffb50e)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

(cherry picked from commit c74027d9d34711c2c4baed7c98bc475d95097be0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2019-10-01 17:16:55 -07:00
Inigo Goiri
2a3eb04996 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb89f4d1edbb54360622adc61ffacbde3)
(cherry picked from commit e59ced9c60a7007551ee8a9f83ce8e266e4dbae1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
 2019-10-01 16:56:29 -07:00
Inigo Goiri
528dc8199b HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 
(cherry picked from commit 865c3289308327788f3bed355864c510deb40956)
 2019-10-01 09:53:31 -07:00
Inigo Goiri
23eed7b201 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 
(cherry picked from commit 0e79a865822eed05f3f8433976b2cfef8f427f25)
(cherry picked from commit fc8a7a9e5b086b1d385fe01207deda000dfcdb57)
 2019-10-01 09:35:14 -07:00
Inigo Goiri
7a915a0ba3 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 
(cherry picked from commit be6c8014e66be919388269b70cb2966c35b8c578)
(cherry picked from commit 06b2eceb76fed768b4a48ca6dd5bccb9864740ed)
 2019-10-01 09:24:21 -07:00
Konstantin V Shvachko
8fc4e406ad HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 17:24:15 -07:00
Konstantin V Shvachko
bece1a1f70 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f6e96c332743a39f965de7995b67bde.
 2019-09-30 17:24:15 -07:00
Surendra Singh Lilhore
1030ac85bb HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar. 
(cherry picked from commit 41e18feda3f5ff924c87c4bed5b5cbbaecb19ae1)
(cherry picked from commit b93b127956508072904b44098fdc1c0dfc899606)
 2019-09-30 16:47:08 -07:00
Ayush Saxena
8cc702ad38 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-28 08:30:26 +05:30
Zhankun Tang
670d0657b4 Make upstream aware of 3.1.3 release. 2019-09-27 16:58:36 +08:00
Chen Liang
3d7730266a HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:59:08 -07:00
Ayush Saxena
5c058549b7 HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:22:11 +05:30
Chen Liang
d25b3339ed HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 17:20:05 -07:00
Kihwal Lee
2661c32c06 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 1851d06eb3b70f39f3054a7c06f0ad2bc664aaec)
 2019-09-18 16:45:29 -05:00
Wei-Chiu Chuang
0fcbeed49a HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 5dd859a8a0d038ff04acbb96741a215d6a0e8c29)
(cherry picked from commit b906f53b0d0e330d65a36295bfb8fe66d0a3995c)
 2019-09-18 09:36:55 -07:00
Anu Engineer
eb5c4ce432 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.

(cherry picked from commit 8aaf5e1a14e577a7d8142bc7d49bb94014032afd)
 2019-09-14 01:35:12 +09:00
Ayush Saxena
cab81f7f5f HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-09-14 10:20:49 +05:30
Takanobu Asanuma
8d5460a045 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 
(cherry picked from commit c255333e20c9af6166db5931d70151011d540359)
 2019-09-11 17:11:13 +09:00
Xiaoyu Yao
ed253c4412
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 16:52:09 -07:00
Xiaoyu Yao
fbdfb36fdf
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit 29e6a97faa81f6b143851b0a038706bccc223130.
 2019-09-05 16:52:09 -07:00
Zhankun Tang
ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Xiaoyu Yao
29e6a97faa
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:58:59 -07:00
Stephen O'Donnell
63b18d764a HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 915cbc91c0a12cc7b4d3ef4ea951941defbbcb33)
(cherry picked from commit 2c7f49743fba587e32398c81412db9eac426cf98)
 2019-09-02 09:58:48 -07:00
Wei-Chiu Chuang
8ffaef78da Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit d42b48d4d0aa08f7570c0f3a80e7fbf58833a6b2.
 2019-09-02 09:58:32 -07:00
Istvan Fajth
e3c01e174c HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-30 13:13:48 -07:00
Ayush Saxena
e44693953d HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:59:52 +05:30
leosunli
0011136e66 Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit 915271245b9a3f682409db380eb311ffd465b041)
(cherry picked from commit b516d2ea7b2b96a1af350edd7964d3e082ee4fe7)
 2019-08-29 18:46:25 -07:00
Stephen O'Donnell
d42b48d4d0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7bebad61d9c3dbff81fdcf243585fd3e9ae59dde)
(cherry picked from commit 9c0d6e16573a055665318bce8f7d38a92c897393)
 2019-08-29 18:26:14 -07:00
Jonathan Hung
79364a9c53 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 
(cherry picked from commit 8ab7020e641e65deb002a10732d23bb22802c09d)
(cherry picked from commit d255efa5f8dc57c1411ec2fe040bd22516393edd)
 2019-08-27 10:31:21 -07:00
Chen Liang
d2bb764ffa HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:29:23 -07:00
Erik Krogen
9dc921f5e5 HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 
(cherry picked from d699022fce756d25956d33e022100111aa0dd22e)
(cherry picked from 2408c2491f4507f92e4051b72eb0800e5e11f069)
 2019-08-23 11:48:51 -07:00
Chen Liang
0839cf1d4e HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
ef9b0a1d9a HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
3135cc35e3 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-23 10:11:22 -07:00
Chen Liang
3835a6c4e8 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Akira Ajisaka
83f67cca60
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 
(cherry picked from commit bd7baea5a5d4ff351645e34c0ef09b7ba82f4285)
 2019-08-23 15:29:25 +09:00
Akira Ajisaka
b40aa71cf0
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 
(cherry picked from commit abc8fde4caea0e197568ee28392c46f1ce0d42e1)
 2019-08-23 15:08:34 +09:00
Xiao Chen
a5690b29a7
HDFS-13642. Creating a file with block size smaller than EC policy's cell size should fail. 
(cherry picked from commit cf4108313da83e28d07676078a33016ec8856ff6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
 2019-08-23 15:03:57 +09:00
Ayush Saxena
900384cf57 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 22:54:50 +05:30
Wei-Chiu Chuang
2ce0914176 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit e978c6c9ed83301fcf8359c35e74ed68045c1a61.
 2019-08-21 09:00:59 -07:00
Surendra Singh Lilhore
c2aeeb01fa HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f95988113da3f06f6d975f99f1ee51d88a793537)
(cherry picked from commit 03c62c7989f818c49d5afca0ac94c7e72a091066)
 2019-08-21 09:00:52 -07:00
Siyao Meng
e0a5fee7f5 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1291) Contributed by Siyao Meng. 2019-08-20 10:49:51 -07:00
Wei-Chiu Chuang
7a1f508f58 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
(cherry picked from commit f78dcbaae5535e3e05c9f29986b8963ba262baa9)
 2019-08-20 10:36:55 -07:00
Wei-Chiu Chuang
d8c60f829f HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde289719e0b300e11568c5e68c36b5d05)
(cherry picked from commit b004d305e933bdb588b84062be87c23168640697)
 2019-08-20 10:34:25 -07:00
Wei-Chiu Chuang
fc062b6fef HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342f3c8cb8246f011abb9bcb0b6ef3eaa)
(cherry picked from commit 6d93886231112c93218066008f957984def3b43b)
 2019-08-19 13:11:21 -07:00