Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,349 Commits 326 Branches 371 Tags
Commit Graph

19349 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Arpit Agarwal
8bb98076be HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 
(cherry picked from commit a64a43b77fb1032dcb66730a6b6257a24726c256)
(cherry picked from commit 8b1238171752d03712ae69d8464108ef0803ae10)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
 2019-10-04 11:23:03 -07:00
Erik Krogen
56caacac1f HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa2ef3cd9dd54a6c5009029721e030ed)
(cherry picked from 69b0c513a9b11cc7f795747732173b36aacbe794)
 2019-10-04 10:48:17 -07:00
Siyao Meng
61dc877b00 HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b384af93726e4f2e6f5e28c6e3a8f65)
 2019-10-04 09:19:30 -07:00
Ayush Saxena
8ce5015e94 HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46)
(cherry picked from commit e3f54a7babf942efbe879aabef65a5b6df57fb65)
 2019-10-04 08:52:03 -07:00
Takanobu Asanuma
c48f2730b4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085dbc561c785cef1613bacaf6735d909)
 2019-10-04 16:49:58 +09:00
Stephen O'Donnell
71dcb8f6d4 HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167ff9566012ef32104a3964887d62c899)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java

(cherry picked from commit 966193153f9cd75e009c8db3502e1b3ba2cdfa25)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
 2019-10-03 22:28:22 -07:00
Tsz Wo Nicholas Sze
ce804be72e HADOOP-16127. In ipc.Client, put a new connection could happen after stop. 
(cherry picked from commit 9192f71e21847ad86bc9ff23847d8957dfe8ae58)
 2019-10-03 18:28:13 -07:00
Inigo Goiri
bb1015719c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b227495d748abe8884eee34db507476cee1)
 2019-10-03 18:21:32 -07:00
Wei-Chiu Chuang
aa5dfb2f8a HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 
(cherry picked from commit b94eba9f11af66b10638dd255c224e946d842b8c)
 2019-10-03 17:30:00 -07:00
Inigo Goiri
af688579a6 HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff54f79de0e6439d6c77c2ed437989f10)
 2019-10-03 17:27:52 -07:00
Shashikant Banerjee
2eb5865ad1 HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a53757f8081e8224e0d4b557fe6a0e2)
 2019-10-03 17:26:13 -07:00
Vinayakumar B
c41791671c HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a5305bea1a40c87a4abc0793802bea5c75)
 2019-10-03 17:15:38 -07:00
Vinayakumar B
1a7cb7aba1 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 
(cherry picked from commit 39dc7345b80e27ba8bd1ff4c19ca241aef5ac0fc)
(cherry picked from commit abe14d32d41b4a2bf630e8a89d794d729cee119b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
 2019-10-03 16:59:00 -07:00
Shweta Yakkali
e68ccca2d7 HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:22:00 -07:00
Ayush Saxena
7ca418c2c0 HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:22:00 -07:00
Shweta Yakkali
8ee3a8768d HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 16:22:00 -07:00
Brahma Reddy Battula
ddea9d68a0 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 15d8f592b51d08a658e88c6a7a7596e0edf5793b)
(cherry picked from commit 7e10dd03b5e1a3f22b55a3b970239611e6f078c0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2019-10-03 16:16:23 -07:00
Kitti Nanasi
aeb3ae1c4c HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc92009fec2f0ab341f3d878e1b8864b8ea9)
 2019-10-03 15:59:41 -07:00
Ayush Saxena
12f2d121ca HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 
(cherry picked from commit c4c8d5fd0e3c17ccdcf18ece8e005f510328b060)
(cherry picked from commit 9a9edfa87ebb007b1dda089a166184fbb6797820)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
 2019-10-03 15:54:39 -07:00
Giovanni Matteo Fumarola
76c30e4f3c HADOOP-16625: Backport HADOOP-14624 to branch-3.1. 
(cherry picked from commit 79c97f6a0bebc95ff81a8ef9b07d3619f05ed583)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
 2019-10-03 15:45:23 -07:00
Inigo Goiri
0160e2c97f HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 
(cherry picked from commit 944adc61b1830388d520d4052fc7eb6c7ba2790d)
(cherry picked from commit 6148acebc330cd85959534290acb735f0b65381a)
 2019-10-03 15:44:56 -07:00
Surendra Singh Lilhore
ef9439679d HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 
(cherry picked from commit 4852a90e4b077ece2d68595210e62959a9923683)
 2019-10-03 23:57:32 +05:30
HUAN-PING SU
6d2888a233 HADOOP-16542. Update commons-beanutils version to 1.9.4. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 38c1a10024476ae78975e4dc7d27a1524722b79d)
(cherry picked from commit f2be3e92b1fb0c29d306ec2be78732473656b359)
 2019-10-02 22:01:14 -07:00
Chen Liang
40c772a750 HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-10-02 21:17:29 -07:00
Surendra Singh Lilhore
7f882570d7 HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f070bbfe65c736f6f3eca721b9d227f5)
(cherry picked from commit 2e939515dfbaf26ca466c8a755cedde0ce4e9c1a)
 2019-10-02 19:50:45 -07:00
LeonGao91
a989779f2d HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 
(cherry picked from commit 9a1d8cfaf50ec29ffb2d8522ba2f4bc6605d8b8b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

(cherry picked from commit 55e40227d1d20bf88448e213937f6e3166452dc9)
 2019-10-02 17:35:25 -07:00
Chen Liang
460ef89210 HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:40:01 -07:00
Tsz Wo Nicholas Sze
8d71b508ab HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections. 
(cherry picked from commit 0edb0c51dc2c4ae2f353e260f01912e28033d70f)
 2019-10-02 16:32:40 -07:00
Wei-Chiu Chuang
41c0757a1f HDFS-14523. Remove excess read lock for NetworkToplogy. Contributed by Wu Weiwei. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Chen Liang <cliang@apache.org>
(cherry picked from commit 971a4c8e8328a4bdea65de4a0e84c82b5b2de24b)
 2019-10-02 16:27:00 -07:00
paulward24
ab7ecd6a0f HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 
(cherry picked from commit d203045c3024b134d7a0417d1ea3a60d03a1534a)
 2019-10-02 15:07:34 -07:00
paulward24
d95dd33ea1 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 
(cherry picked from commit d8bac50e12d243ef8fd2c7e0ce5c9997131dee74)
(cherry picked from commit 76a91359475444cb6d415e5056430c0e743b88ee)
 2019-10-02 14:31:23 -07:00
Ayush Saxena
122b02e288 HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 
(cherry picked from commit 66400c1cbb2b4b2f08f7db965c8b7237072bdcc4)
(cherry picked from commit 2bb4ac18ec0ea8804b424b593615bb06e21a4b6c)
 2019-10-02 13:49:20 -07:00
Surendra Singh Lilhore
f0bdab02e7 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 
(cherry picked from commit d1c303a49763029fffa5164295034af8e81e74a0)
(cherry picked from commit eb1ddcd04c9b0457e19fcc3b320d5b86cc1fda64)
 2019-10-02 13:41:23 -07:00
Inigo Goiri
f1fe3abac9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0cc7d8528e9b4411457f04a528601c69)
 2019-10-01 17:33:08 -07:00
Yiqun Lin
3aea2bcf12 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee871df1111ff77d0f6921b13c8ffb50e)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

(cherry picked from commit c74027d9d34711c2c4baed7c98bc475d95097be0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2019-10-01 17:16:55 -07:00
Inigo Goiri
2a3eb04996 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb89f4d1edbb54360622adc61ffacbde3)
(cherry picked from commit e59ced9c60a7007551ee8a9f83ce8e266e4dbae1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
 2019-10-01 16:56:29 -07:00
Ranith Sardar
a400f396a6 HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 
Contributed by Ranith Sardar.

(cherry picked from commit 546c5d70efebb828389f609a89b123c4ee51f867)
(cherry picked from commit c5eca3f7ee095d6a261eb411ad97aba654d67d13)

 Conflicts:
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
 2019-10-01 16:34:00 -07:00
Stephen O'Donnell
9dfb26136b HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

Signed-off-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit 686c0141eff0886c285b8e52fddade43c1ce4570)
 2019-10-01 13:13:59 -07:00
Inigo Goiri
528dc8199b HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 
(cherry picked from commit 865c3289308327788f3bed355864c510deb40956)
 2019-10-01 09:53:31 -07:00
Surendra Singh Lilhore
5436df0c94 HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena. 
(cherry picked from commit dacc1a759e3ba3eca000cbacc6145b231253b174)
 2019-10-01 09:47:11 -07:00
Inigo Goiri
23eed7b201 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 
(cherry picked from commit 0e79a865822eed05f3f8433976b2cfef8f427f25)
(cherry picked from commit fc8a7a9e5b086b1d385fe01207deda000dfcdb57)
 2019-10-01 09:35:14 -07:00
Inigo Goiri
7a915a0ba3 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 
(cherry picked from commit be6c8014e66be919388269b70cb2966c35b8c578)
(cherry picked from commit 06b2eceb76fed768b4a48ca6dd5bccb9864740ed)
 2019-10-01 09:24:21 -07:00
Wei-Chiu Chuang
57ed0ff225 HADOOP-15418. Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException. Contributed by lqjack and Suma Shivaprasad 
(cherry picked from commit cd2158456db8c89eeea64b72654a736ea8607e23)
(cherry picked from commit ab3fb28d47e8fa260d2de7490aea59404d571c2e)
 2019-10-01 09:03:01 -07:00
Kihwal Lee
757c11e628 HADOOP-15909. KeyProvider class should implement Closeable. Contributed 
by Kuhu Shukla.

(cherry picked from commit 33c009a45d95c9b582db25acd205d087e01e2085)
 2019-10-01 08:53:47 -07:00
Sunil G
933d81f0ce YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 
(cherry picked from commit 99cd7572f113cbbb8f5ef89af60af3734fe8cfa7)
 2019-10-01 20:08:12 +05:30
Sunil G
71792f2122 YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 20:03:35 +05:30
bibinchundatt
6529a30d9e YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:06:38 +05:30
Erik Krogen
5e56914650 HADOOP-16531. Log more timing information for slow RPCs. Contributed by Chen Zhang. 
(cherry picked from commit a23417533e1ee052893baf207ec636c4993c5994)
 2019-09-30 17:35:38 -07:00
Wei-Chiu Chuang
72ef752876 HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko. 
(cherry picked from commit e872ceb810a343da7fce7185dca78d3b9aad9b7b)
(cherry picked from commit 46e72775f5b2156f49b15beac8bffff9cd5aad7f)
 2019-09-30 17:32:23 -07:00
Konstantin V Shvachko
8fc4e406ad HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 17:24:15 -07:00