Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,193 Commits 326 Branches 371 Tags
Commit Graph

21193 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Szilard Nemeth
42177e8b78 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 
(cherry picked from commit 6f1ab9516869dbfdee034f4a17bccc379a39fc31)
 2019-10-09 13:28:01 +02:00
Szilard Nemeth
0ddb48a303 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 
(cherry picked from commit 35f093f5b3f310af49a8bb476e62173862f5af51)
 2019-10-09 13:12:03 +02:00
Sunil G
9cb3ab058f YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit be901f49628a553cfe6a3942478cb528cce2b266)
 2019-10-09 15:54:09 +05:30
Chen Liang
a12ab911ee HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a6695f45f0d9cea5ae9aae83941d360)
 2019-10-08 12:03:04 -07:00
Hrishikesh Gadre
aec58731a8 HADOOP-16491. Upgrade jetty version to 9.3.27. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3cc0ace2037ee039fc2e97d5737e92d3cb64cc72)
 2019-10-08 11:29:47 -07:00
Jonathan Hung
ad3c98456d YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 
(cherry picked from commit eebd313d76ed742fe82292bd8c0184970cdc5692)
 2019-10-08 11:17:05 -07:00
Szilard Nemeth
9b4aba49d1 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-10-08 17:41:46 +02:00
Ayush Saxena
57ab854492 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:22:16 +05:30
Surendra Singh Lilhore
a0cdebc409 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 382967be51052d59e31d8d05713645b8d3c2325b)
 2019-10-08 00:26:27 +05:30
Sunil G
5704f15589 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 3a0afcfb7f196bee8e534a3e80211bfa077b2aee.
 2019-10-05 09:16:04 +05:30
He Xiaoqiao
e29ae7db12 HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37bffb77da17c025f0b9b673d7088bc6)
 2019-10-04 12:07:30 -07:00
He Xiaoqiao
80392e94b6 HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e94442825fe286c2b18518925d980e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2019-10-04 12:06:55 -07:00
Erik Krogen
6630c9b75d HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e0014343f60f853cb796781ca1fa03a72efd)
 2019-10-04 11:15:49 -07:00
Erik Krogen
69b0c513a9 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa2ef3cd9dd54a6c5009029721e030ed)
 2019-10-04 10:48:33 -07:00
Eric Yang
8bb2b00d38 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0e9812d1f787efb3c40155b0fdeed9c)
 2019-10-04 13:16:08 -04:00
Siyao Meng
673c9d53ca HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b384af93726e4f2e6f5e28c6e3a8f65)
 2019-10-04 09:40:22 -07:00
Sunil G
3a0afcfb7f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit 4510970e2f7728d036c750b596985e5ffa357b60)
 2019-10-04 21:49:49 +05:30
Ayush Saxena
702572434c HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46)
 2019-10-04 08:52:59 -07:00
Takanobu Asanuma
732a68cfb4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085dbc561c785cef1613bacaf6735d909)
 2019-10-04 16:48:50 +09:00
Stephen O'Donnell
966193153f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167ff9566012ef32104a3964887d62c899)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java
 2019-10-03 22:18:29 -07:00
Tsz Wo Nicholas Sze
8acd9ab695 HADOOP-16127. In ipc.Client, put a new connection could happen after stop. 
(cherry picked from commit 9192f71e21847ad86bc9ff23847d8957dfe8ae58)
 2019-10-03 18:28:21 -07:00
Inigo Goiri
94af96325c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b227495d748abe8884eee34db507476cee1)
 2019-10-03 18:21:46 -07:00
Wei-Chiu Chuang
6facb3f7c6 HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 
(cherry picked from commit b94eba9f11af66b10638dd255c224e946d842b8c)
 2019-10-03 17:30:08 -07:00
Inigo Goiri
21a89d544f HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff54f79de0e6439d6c77c2ed437989f10)
 2019-10-03 17:27:44 -07:00
Shashikant Banerjee
f14fb9081f HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a53757f8081e8224e0d4b557fe6a0e2)
 2019-10-03 17:27:06 -07:00
Vinayakumar B
4e223d9968 HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a5305bea1a40c87a4abc0793802bea5c75)
 2019-10-03 17:15:45 -07:00
Kitti Nanasi
cb701f4da0 HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc92009fec2f0ab341f3d878e1b8864b8ea9)
 2019-10-03 16:18:23 -07:00
Shweta Yakkali
eaf09e51ba HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:13:44 -07:00
Ayush Saxena
ebd7a0f155 HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:00:38 -07:00
Shweta Yakkali
3186afa24c HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 15:58:00 -07:00
Ayush Saxena
9a9edfa87e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 
(cherry picked from commit c4c8d5fd0e3c17ccdcf18ece8e005f510328b060)
 2019-10-03 14:56:09 -07:00
HUAN-PING SU
f2be3e92b1 HADOOP-16542. Update commons-beanutils version to 1.9.4. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 38c1a10024476ae78975e4dc7d27a1524722b79d)
 2019-10-02 22:00:40 -07:00
Chen Liang
05a1eb788f HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-10-02 21:12:54 -07:00
LeonGao91
885dd17595 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 
(cherry picked from commit 9a1d8cfaf50ec29ffb2d8522ba2f4bc6605d8b8b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
 2019-10-02 17:35:33 -07:00
Chen Liang
8a866f01c9 HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:39:26 -07:00
Tsz Wo Nicholas Sze
824f717882 HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections. 
(cherry picked from commit 0edb0c51dc2c4ae2f353e260f01912e28033d70f)
 2019-10-02 16:32:53 -07:00
Wei-Chiu Chuang
28c11ff154 HDFS-14523. Remove excess read lock for NetworkToplogy. Contributed by Wu Weiwei. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Chen Liang <cliang@apache.org>
(cherry picked from commit 971a4c8e8328a4bdea65de4a0e84c82b5b2de24b)
 2019-10-02 16:27:07 -07:00
paulward24
0559c1dba3 HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 
(cherry picked from commit d203045c3024b134d7a0417d1ea3a60d03a1534a)
 2019-10-02 15:07:40 -07:00
paulward24
76a9135947 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 
(cherry picked from commit d8bac50e12d243ef8fd2c7e0ce5c9997131dee74)
 2019-10-02 14:31:13 -07:00
Inigo Goiri
6148acebc3 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 
(cherry picked from commit 944adc61b1830388d520d4052fc7eb6c7ba2790d)
 2019-10-02 13:55:47 -07:00
Ayush Saxena
2bb4ac18ec HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 
(cherry picked from commit 66400c1cbb2b4b2f08f7db965c8b7237072bdcc4)
 2019-10-02 13:49:12 -07:00
Surendra Singh Lilhore
7d7330aff0 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 
(cherry picked from commit 4852a90e4b077ece2d68595210e62959a9923683)
 2019-10-02 13:39:40 -07:00
Inigo Goiri
d2a7a128d9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0cc7d8528e9b4411457f04a528601c69)
 2019-10-01 17:33:14 -07:00
Yiqun Lin
c74027d9d3 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee871df1111ff77d0f6921b13c8ffb50e)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
 2019-10-01 17:14:46 -07:00
Inigo Goiri
3510a9e5e5 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb89f4d1edbb54360622adc61ffacbde3)
 2019-10-01 16:57:20 -07:00
Inigo Goiri
917e53f066 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 
(cherry picked from commit 865c3289308327788f3bed355864c510deb40956)
 2019-10-01 09:53:36 -07:00
Surendra Singh Lilhore
a131611b4b HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena. 
(cherry picked from commit dacc1a759e3ba3eca000cbacc6145b231253b174)
 2019-10-01 09:47:18 -07:00
Inigo Goiri
fc8a7a9e5b HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 
(cherry picked from commit 0e79a865822eed05f3f8433976b2cfef8f427f25)
 2019-10-01 09:35:04 -07:00
Inigo Goiri
06b2eceb76 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 
(cherry picked from commit be6c8014e66be919388269b70cb2966c35b8c578)
 2019-10-01 09:22:15 -07:00
Wei-Chiu Chuang
ab3fb28d47 HADOOP-15418. Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException. Contributed by lqjack and Suma Shivaprasad 
(cherry picked from commit cd2158456db8c89eeea64b72654a736ea8607e23)
 2019-10-01 09:02:48 -07:00