Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,275 Commits 326 Branches 371 Tags
Commit Graph

19275 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wei-Chiu Chuang
f33bcbb687 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 
(cherry picked from commit a5d76a4965b686ec5532add511c64d5f87c88865)
 2019-06-20 20:54:43 -07:00
Takanobu Asanuma
2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)
 2019-06-20 20:18:00 -07:00
Wei-Chiu Chuang
e4a6976c30 HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 840d02ca5ba013bddd28d7c1abd242b5bb8cc39f)
 2019-06-20 19:58:28 -07:00
Inigo Goiri
3dd4000884 HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke. 
(cherry picked from commit f4e18242bd8117a5c506ec6d3f25c85011fa82d0)
 2019-06-20 13:38:48 -07:00
Wei-Chiu Chuang
58c1920472 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 49587e4df3e955466670384de867f690b557fe6e.
 2019-06-19 06:55:25 -07:00
Virajith Jalaparti
49587e4df3 HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
(cherry picked from commit 36dc0810bf8032e4741e27a8f2b00e64dbdaef2a)
 2019-06-18 22:17:08 -07:00
Zsolt Venczel
4ea3b04bf3 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff6bf8d5e1d05748ba32b679e397b53f)
 2019-06-18 14:38:05 -07:00
Elek, Marton
293f8bbd84 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
(cherry picked from commit 1a99f9794519faee5a196ab3d9cb91d55bfd0002)
 2019-06-18 12:00:48 -07:00
Wei-Chiu Chuang
29c4aded34 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
(cherry picked from commit 223a582eb839bb6e4298a2fe0243bc27d73fc977)
 2019-06-18 00:34:55 -07:00
Shweta Yakkali
0b3a2eb8a7 HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd4a7633ece11c528a58146f92522b55e7be4dc6)
(cherry picked from commit 99abd35a75fc29d89a03aaf9f1919e9d48f4abec)
 2019-06-17 23:48:40 -07:00
Adam Antal
15689ce0e8 HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 9f47a2e4a9845b84cc963708c7924b92df2964f6)
(cherry picked from commit 26671772486b71f27ba37c836006b4ab05fd2b96)
 2019-06-17 22:56:18 -07:00
Wei-Chiu Chuang
4362fcb629 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 
(cherry picked from commit 098c325a78dce2697ec94a22a4a3981fa5a3849e)
(cherry picked from commit f5428d4062c0a5e98461ea3b8420dc68555681f9)
 2019-06-17 22:48:07 -07:00
Zhankun Tang
829202740a YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji. 
(cherry picked from commit 67414a1a80039e70e0afc1de171831a6e981f37a)
 2019-06-18 13:20:07 +08:00
Wei-Chiu Chuang
8f4204dc3b HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor. 
(cherry picked from commit f1c239c6a4c26e9057373b9b9400e54083290f65)
(cherry picked from commit e8db39b0e9ad66af3dada1ec7ad06b7d06f8e9f8)
 2019-06-17 21:56:04 -07:00
Weiwei Yang
3bdb661f35 HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
(cherry picked from commit f36cdc4d7f45d1a3f8edd2d804c38b0b39990410)
 2019-06-17 21:47:07 -07:00
Wei-Chiu Chuang
fc8983a5b8 HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)
(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)
 2019-06-17 20:20:08 -07:00
Akira Ajisaka
3d1d8623a1 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827381a7f87c01f0befc2475f9478e63b)
(cherry picked from commit f68099aa1316f34591c757bcf4198a3a3c7f691a)
 2019-06-17 17:04:07 -07:00
Wei-Chiu Chuang
b3e0430c3b HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
(cherry picked from commit 0eb42e1e05b780dab718bb3adfdd55f527dd5545)
 2019-06-17 15:18:39 -07:00
huzheng
b082628e5a HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)
(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)
 2019-06-17 07:21:07 -07:00
Weiwei Yang
56a4935048 YARN-9621. Fix TestDSWithMultipleNodeManager.testDistributedShellWithPlacementConstraint on branch-3.1. Contributed by Prabhu Joseph. 2019-06-17 17:22:49 +08:00
Inigo Goiri
76a66998fe HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:28:38 -07:00
Takanobu Asanuma
94e66ba338 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 
(cherry picked from commit 54f9f75a443d7d167a7aa7d04a87e3f5af049887)
 2019-06-14 10:19:50 +09:00
Sean Mackrory
fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Stephen O'Donnell
a9b8310584 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0f14526bfb7052f2a2e64543dc0fcbc)
(cherry picked from commit 71159f9540d5a2aac6fb904649639f75e457e94d)
 2019-06-12 10:31:21 -07:00
Sunil G
c343554e2a YARN-9543. [UI2] Handle ATSv2 server down or failures cases gracefully in YARN UI v2. Contributed by Zoltan Siegl and Akhil P B. 
(cherry picked from commit 52128e352a30b70b83483f9290d9e94e98929705)
 2019-06-12 19:28:43 +05:30
Sunil G
bc028d3ebb YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 
(cherry picked from commit 72203f7a12c943ca231fbc40c058a1a094b009cd)
 2019-06-12 19:28:10 +05:30
Sunil G
1bb9e9a4f2 Revert "YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl." 
This reverts commit d65371c4e8bbb4ae655ccacda389cd37a18fab32.
 2019-06-12 19:27:21 +05:30
Siyao Meng
d84c45461e HADOOP-16263. Update BUILDING.txt with macOS native build instructions. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 4fecc2a95e2bd7a4f5ba0b930f1bd6be7227d1b5)
(cherry picked from commit f4afeb680548643c4b194b75b421cd9ca930afc5)
 2019-06-11 15:06:27 -07:00
bibinchundatt
f42e246f8a YARN-9547. ContainerStatusPBImpl default execution type is not returned. Contributed by Bilwa S T. 
(cherry picked from commit 3303723f55edc6ef55a07707c4453395e82ff060)
 2019-06-11 23:43:54 +05:30
bibinchundatt
d386f595f9 YARN-9565. RMAppImpl#ranNodes not cleared on FinalTransition. Contributed by Bilwa S T. 
(cherry picked from commit 60c95e9b6a899e37ecdc8bce7bb6d9ed0dc7a6be)
 2019-06-11 23:15:02 +05:30
bibinchundatt
4a39165b41 YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 
(cherry picked from commit 6d80b9bc3ff3ba8073e3faf64551b9109d2aa2ad)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
 2019-06-11 23:05:06 +05:30
He Xiaoqiao
03ec4c4c7c HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9244f3f3474d316255ac98717ed5719)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit d7560c866ecbfcd533a43890522ccb8b732a5382)
 2019-06-10 17:27:48 -07:00
Hanisha Koneru
dd795edcc7 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40dd99068a3f3b155261b1dac6c67828)
(cherry picked from commit 210480a23cab052c5ea538fbfd9e50d888b8d3ff)
 2019-06-10 13:47:17 -07:00
Erik Krogen
cf3c1b9ffc HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 
(cherry-picked from 76b94c274fe9775efcfd51c676d80c88a4f7fdb9)
(cherry-picked from f607efd1950211f85eb571564b774184f8fa1687)
 2019-06-07 14:22:30 -07:00
Sunil G
d65371c4e8 YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 
(cherry picked from commit f1d3a17d3e67ec2acad52227a3f4eb7cca83e468)
 2019-06-06 06:25:02 +05:30
Sunil G
6be665cfc6 YARN-8906. [UI2] NM hostnames not displayed correctly in Node Heatmap Chart. Contributed by Akhil PB. 
(cherry picked from commit 59719dc560cf67f485d8e5b4a6f0f38ef97d536b)
 2019-06-03 15:54:33 +05:30
Sunil G
894ef5c07b YARN-8947. [UI2] Active User info missing from UI2. Contributed by Akhil PB. 
(cherry picked from commit 7f46dda513fb79c349acb73bdb90b689df9cc18d)
 2019-06-03 12:25:40 +05:30
Weiwei Yang
23f9508a89 YARN-9507. Fix NPE in NodeManager#serviceStop on startup failure. Contributed by Bilwa S T. 
(cherry picked from commit 4530f4500d308c9cefbcc5990769c04bd061ad87)
 2019-06-03 14:26:16 +08:00
Kihwal Lee
a935258e8c HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)
 2019-05-30 17:31:30 -05:00
Wanqiang Ji
2133bb561d
HADOOP-16334. Fix yetus-wrapper not working when HADOOP_YETUS_VERSION >= 0.9.0 
This closes #864

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d23e8e9232f030fcdeff91b9e8a625516244dc72)
 2019-05-30 15:13:20 +09:00
Ayush Saxena
90cc5ddd49 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)
(cherry picked from commit 2cd7577a5a58db42a14a659b85fbd475091d0cb2)
 2019-05-29 20:58:34 -07:00
Eric Yang
413a6b63bc YARN-9542. Fix LogsCLI guessAppOwner ignores custome file format suffix. 
Contributed by Prabhu Joseph

(cherry picked from commit b2a39e8883f8128e44543c2279dcc1835af72652)
 2019-05-29 18:05:47 -04:00
Eric E Payne
9c3ab58aa7 YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph. 
(cherry picked from commit 3c63551101dc17038efb9f5345833d2883fce86e)
 2019-05-29 19:08:27 +00:00
Ahmed Hussein
f2202f7990 YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit abf76ac371e4611c8eb371736b433e3d89c9d2ae)
 2019-05-29 12:47:27 -05:00
Takanobu Asanuma
8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka
f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Erik Krogen
9295e95174 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f002f74cb4a0360b6682a1b6796d1598)
(cherry-picked from c26cf22d6b3949a4798d8133669b451660a0f050)
 2019-05-24 13:19:39 -07:00
Konstantin V Shvachko
799ca572f7 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:35:32 -07:00
Shweta
a6ff20d25d HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93d7630062247793a4860704fad4a31437693de1)
(cherry picked from commit 7a5d9ab9eed194fabe609def69efe41bea4ed1f6)
 2019-05-24 18:43:49 +02:00
Wanqiang Ji
6690955358
MAPREDUCE-7205. Treat container scheduler kill exit code as a task attempt killing event. 
This closes #821

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 67f9a7b165edecbec7c8063758202be4d8cff0f5)
 2019-05-22 14:41:20 +09:00