Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,338 Commits 326 Branches 371 Tags
Commit Graph

19338 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Masatake Iwasaki
02ae1d6071 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec78c8dab9111f3ba2709cef20794b73a)
 2019-07-04 23:54:18 +09:00
Sunil G
9eb96b0fbf YARN-9644. First RMContext object is always leaked during switch over. Contributed by Bibin A Chundatt. 
(cherry picked from commit d18986e4e89bd4bb3600e95b4690fd32f54a41e5)
 2019-07-04 11:06:41 +05:30
Szilard Nemeth
46177ade8b YARN-9629. Support configurable MIN_LOG_ROLLING_INTERVAL. Contributed by Adam Antal. 
(cherry picked from commit a2a8be18cb5e912c8de0ea6beec1de4a99de656b)
 2019-07-03 14:24:53 +02:00
Inigo Goiri
3ab475377a HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena. 
(cherry picked from commit aa9f0850e85203b2ce4f4a8dc8968e9186cdc67a)
 2019-07-02 13:26:17 -07:00
Sunil G
d2a5749482 YARN-9327. Improve synchronisation in ProtoUtils#convertToProtoFormat block. Contributed by Bibin A Chundatt. 
(cherry picked from commit 0c8813f135f8c17f88660bb92529c15bb3a157ca)
 2019-07-02 12:15:40 +05:30
Weiwei Yang
46b81a982b YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. Contributed by hunshenshi. 
(cherry picked from commit 570eee30e5ab5cf37b1a758934987cbf61140f6a)
 2019-07-02 10:17:56 +08:00
bibinchundatt
4f622ecad8 YARN-9639. DecommissioningNodesWatcher cause memory leak. Contributed by Bilwa S T. 
(cherry picked from commit be80334cdf255616f589217726483194fb56dcc6)
 2019-06-27 10:11:30 +05:30
Anu Engineer
1bae5c7024 HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.

(cherry picked from commit 041e7a7dee4a17714f31952dc6364c77a65b1b73)
(cherry picked from commit e7fce2104f28c48c817822bed9349d44b0694699)
 2019-06-24 21:20:00 -07:00
Inigo Goiri
90b88db35d HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42a3ef94d6bef160bc58d807f2faa104)
(cherry picked from commit 96371245357bda63b3ede10f37a37f5333a85d69)
 2019-06-24 21:17:04 -07:00
Inigo Goiri
aaf74ea5d7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6fddb7adf60dba846d1f7c8a13b8ec1)
 2019-06-24 19:03:26 -07:00
Santosh Marella
026cc2647d HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-23 20:27:01 -07:00
Wei-Chiu Chuang
529d0955f5 HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 
(cherry picked from commit 645d67bc4f4e29d10ef810386c89e6a7c8c61862)
(cherry picked from commit d5ecca1d4ca2b567167402bf8745fb2417bbf624)
 2019-06-21 18:27:50 -07:00
Wei-Chiu Chuang
4507c2f3a5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 
(cherry picked from commit 1524e2e6c52aba966cbbf1d8025ba165688ab9bb)
(cherry picked from commit 845d7611babac68a2b0caa6e4b8963f0c8332540)
 2019-06-21 18:18:36 -07:00
Wei-Chiu Chuang
f33bcbb687 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 
(cherry picked from commit a5d76a4965b686ec5532add511c64d5f87c88865)
 2019-06-20 20:54:43 -07:00
Takanobu Asanuma
2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)
 2019-06-20 20:18:00 -07:00
Wei-Chiu Chuang
e4a6976c30 HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 840d02ca5ba013bddd28d7c1abd242b5bb8cc39f)
 2019-06-20 19:58:28 -07:00
Inigo Goiri
3dd4000884 HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke. 
(cherry picked from commit f4e18242bd8117a5c506ec6d3f25c85011fa82d0)
 2019-06-20 13:38:48 -07:00
Wei-Chiu Chuang
58c1920472 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 49587e4df3e955466670384de867f690b557fe6e.
 2019-06-19 06:55:25 -07:00
Virajith Jalaparti
49587e4df3 HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
(cherry picked from commit 36dc0810bf8032e4741e27a8f2b00e64dbdaef2a)
 2019-06-18 22:17:08 -07:00
Zsolt Venczel
4ea3b04bf3 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff6bf8d5e1d05748ba32b679e397b53f)
 2019-06-18 14:38:05 -07:00
Elek, Marton
293f8bbd84 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
(cherry picked from commit 1a99f9794519faee5a196ab3d9cb91d55bfd0002)
 2019-06-18 12:00:48 -07:00
Wei-Chiu Chuang
29c4aded34 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
(cherry picked from commit 223a582eb839bb6e4298a2fe0243bc27d73fc977)
 2019-06-18 00:34:55 -07:00
Shweta Yakkali
0b3a2eb8a7 HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd4a7633ece11c528a58146f92522b55e7be4dc6)
(cherry picked from commit 99abd35a75fc29d89a03aaf9f1919e9d48f4abec)
 2019-06-17 23:48:40 -07:00
Adam Antal
15689ce0e8 HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 9f47a2e4a9845b84cc963708c7924b92df2964f6)
(cherry picked from commit 26671772486b71f27ba37c836006b4ab05fd2b96)
 2019-06-17 22:56:18 -07:00
Wei-Chiu Chuang
4362fcb629 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 
(cherry picked from commit 098c325a78dce2697ec94a22a4a3981fa5a3849e)
(cherry picked from commit f5428d4062c0a5e98461ea3b8420dc68555681f9)
 2019-06-17 22:48:07 -07:00
Zhankun Tang
829202740a YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji. 
(cherry picked from commit 67414a1a80039e70e0afc1de171831a6e981f37a)
 2019-06-18 13:20:07 +08:00
Wei-Chiu Chuang
8f4204dc3b HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor. 
(cherry picked from commit f1c239c6a4c26e9057373b9b9400e54083290f65)
(cherry picked from commit e8db39b0e9ad66af3dada1ec7ad06b7d06f8e9f8)
 2019-06-17 21:56:04 -07:00
Weiwei Yang
3bdb661f35 HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
(cherry picked from commit f36cdc4d7f45d1a3f8edd2d804c38b0b39990410)
 2019-06-17 21:47:07 -07:00
Wei-Chiu Chuang
fc8983a5b8 HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)
(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)
 2019-06-17 20:20:08 -07:00
Akira Ajisaka
3d1d8623a1 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827381a7f87c01f0befc2475f9478e63b)
(cherry picked from commit f68099aa1316f34591c757bcf4198a3a3c7f691a)
 2019-06-17 17:04:07 -07:00
Wei-Chiu Chuang
b3e0430c3b HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
(cherry picked from commit 0eb42e1e05b780dab718bb3adfdd55f527dd5545)
 2019-06-17 15:18:39 -07:00
huzheng
b082628e5a HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)
(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)
 2019-06-17 07:21:07 -07:00
Weiwei Yang
56a4935048 YARN-9621. Fix TestDSWithMultipleNodeManager.testDistributedShellWithPlacementConstraint on branch-3.1. Contributed by Prabhu Joseph. 2019-06-17 17:22:49 +08:00
Inigo Goiri
76a66998fe HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:28:38 -07:00
Takanobu Asanuma
94e66ba338 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 
(cherry picked from commit 54f9f75a443d7d167a7aa7d04a87e3f5af049887)
 2019-06-14 10:19:50 +09:00
Sean Mackrory
fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Stephen O'Donnell
a9b8310584 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0f14526bfb7052f2a2e64543dc0fcbc)
(cherry picked from commit 71159f9540d5a2aac6fb904649639f75e457e94d)
 2019-06-12 10:31:21 -07:00
Sunil G
c343554e2a YARN-9543. [UI2] Handle ATSv2 server down or failures cases gracefully in YARN UI v2. Contributed by Zoltan Siegl and Akhil P B. 
(cherry picked from commit 52128e352a30b70b83483f9290d9e94e98929705)
 2019-06-12 19:28:43 +05:30
Sunil G
bc028d3ebb YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 
(cherry picked from commit 72203f7a12c943ca231fbc40c058a1a094b009cd)
 2019-06-12 19:28:10 +05:30
Sunil G
1bb9e9a4f2 Revert "YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl." 
This reverts commit d65371c4e8bbb4ae655ccacda389cd37a18fab32.
 2019-06-12 19:27:21 +05:30
Siyao Meng
d84c45461e HADOOP-16263. Update BUILDING.txt with macOS native build instructions. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 4fecc2a95e2bd7a4f5ba0b930f1bd6be7227d1b5)
(cherry picked from commit f4afeb680548643c4b194b75b421cd9ca930afc5)
 2019-06-11 15:06:27 -07:00
bibinchundatt
f42e246f8a YARN-9547. ContainerStatusPBImpl default execution type is not returned. Contributed by Bilwa S T. 
(cherry picked from commit 3303723f55edc6ef55a07707c4453395e82ff060)
 2019-06-11 23:43:54 +05:30
bibinchundatt
d386f595f9 YARN-9565. RMAppImpl#ranNodes not cleared on FinalTransition. Contributed by Bilwa S T. 
(cherry picked from commit 60c95e9b6a899e37ecdc8bce7bb6d9ed0dc7a6be)
 2019-06-11 23:15:02 +05:30
bibinchundatt
4a39165b41 YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 
(cherry picked from commit 6d80b9bc3ff3ba8073e3faf64551b9109d2aa2ad)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
 2019-06-11 23:05:06 +05:30
He Xiaoqiao
03ec4c4c7c HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9244f3f3474d316255ac98717ed5719)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit d7560c866ecbfcd533a43890522ccb8b732a5382)
 2019-06-10 17:27:48 -07:00
Hanisha Koneru
dd795edcc7 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40dd99068a3f3b155261b1dac6c67828)
(cherry picked from commit 210480a23cab052c5ea538fbfd9e50d888b8d3ff)
 2019-06-10 13:47:17 -07:00
Erik Krogen
cf3c1b9ffc HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 
(cherry-picked from 76b94c274fe9775efcfd51c676d80c88a4f7fdb9)
(cherry-picked from f607efd1950211f85eb571564b774184f8fa1687)
 2019-06-07 14:22:30 -07:00
Sunil G
d65371c4e8 YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 
(cherry picked from commit f1d3a17d3e67ec2acad52227a3f4eb7cca83e468)
 2019-06-06 06:25:02 +05:30
Sunil G
6be665cfc6 YARN-8906. [UI2] NM hostnames not displayed correctly in Node Heatmap Chart. Contributed by Akhil PB. 
(cherry picked from commit 59719dc560cf67f485d8e5b4a6f0f38ef97d536b)
 2019-06-03 15:54:33 +05:30
Sunil G
894ef5c07b YARN-8947. [UI2] Active User info missing from UI2. Contributed by Akhil PB. 
(cherry picked from commit 7f46dda513fb79c349acb73bdb90b689df9cc18d)
 2019-06-03 12:25:40 +05:30