Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,302 Commits 326 Branches 371 Tags
Commit Graph

17302 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Arpit Agarwal
16b2cad8e8 HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. Contributed by Ajay Kumar. 2018-02-13 10:14:29 -08:00
Jason Lowe
bc2d4ba507 MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko 
(cherry picked from commit 87e2570a1419d3616de2de3b553108ad1a8af425)
 2018-02-12 14:49:35 -06:00
Brahma Reddy Battula
055d5e0b85 HDFS-8693. Addendum patch to execute the command using UGI. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 35c17351cab645dcc72e0d2ae1608507aa787ffb)
 2018-02-12 22:17:35 +05:30
Arpit Agarwal
41cf2b8e7c HDFS-10453. ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.. Contributed by He Xiaoqiao. 2018-02-12 07:16:48 -08:00
Yiqun Lin
38febc0ae2 HDFS-13099. RBF: Use the ZooKeeper as the default State Store. Contributed by Yiqun Lin. 
(cherry picked from commit 543f3abbee79d7ec70353f0cdda6397ee001324e)
 2018-02-09 13:59:38 +08:00
Xiaoyu Yao
49ab09f00b HDFS-13120. Snapshot diff could be corrupted after concat. Contributed by Xiaoyu Yao. 
(cherry picked from commit 8faf0b50d435039f69ea35f592856ca04d378809)
 2018-02-08 10:18:35 -08:00
Manoj Govindassamy
bef769e436 HDFS-12985. NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted. 
(cherry picked from commit 73ff09b79a5cf9932edc21c58f3a730f7379086b)

(cherry picked from commit 821729905ed8439ee5f15372faaf6b40aff7f155)
 2018-02-07 19:49:27 -08:00
Yongjun Zhang
5105b60614 HDFS-13115. In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted. Contributed by Yongjun Zhang. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java

(cherry picked from commit 60d82ac61f2afa2a61c8eeabf20e45b5a89311d0)
 2018-02-07 16:49:26 -08:00
Anu Engineer
2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bdf9fb7aa9181e866eea383f89bf171f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb35dbe30c75c648bf759e8aeb715834a)
 2018-02-07 11:16:53 -08:00
Brahma Reddy Battula
c6349c3204 HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up. Contributed by Jianfei Jiang. 
(cherry picked from commit 01bd6ab18fa48f4c7cac1497905b52e547962599)
 2018-02-07 23:27:24 +05:30
Billie Rinaldi
0ec45f73a1 Revert "YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi." 
This reverts commit ef212b8550bd607e9e166a99f76aae4558729464.
 2018-02-07 07:51:56 -08:00
Xiao Chen
49de91446f HDFS-12933. Improve logging when DFSStripedOutputStream failed to write some blocks. Contributed by chencan. 
(cherry picked from commit e5c2fdddd69ceca21d641de56904648acedc74bf)
 2018-02-06 22:57:15 -08:00
Wei-Chiu Chuang
29a3b64ec0 HDFS-11187. Optimize disk access for last partial chunk checksum of Finalized replica. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 2021f4bdce3b27c46edaad198f0007a26a8a1391)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBuilder.java
 2018-02-06 22:57:15 -08:00
Kihwal Lee
d436c403b2 HADOOP-15212. Add independent secret manager method for logging expired tokens. Contributed by Daryn Sharp. 
(cherry picked from commit 2dd960de983a30bf0d9ee957bdb09f825f9d40a3)
 2018-02-06 15:34:52 -06:00
Billie Rinaldi
615bd15072 YARN-7890. NPE during container relaunch. Contributed by Jason Lowe 
(cherry picked from commit a196ee9362a1b35e5de20ee519f7c544ab1588e1)
 2018-02-06 11:34:49 -08:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Jason Lowe
07f779483c YARN-7849. TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error. Contributed by Botong Huang 
(cherry picked from commit 60656bcecadd80e28c81bc943b44abf13d20abae)
 2018-02-05 17:17:15 -06:00
Yongjun Zhang
35247cca87 HDFS-13100. Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. Contributed by Aaron T. Myers and Yongjun Zhang. 
(cherry picked from commit 4e9a59ce16e81b4bd6fae443a997ef24d588a6e8)

(cherry picked from commit 0524fac21ef8a8e440a2fe759936e6fb6a63cbf1)
 2018-02-04 22:53:09 -08:00
Yiqun Lin
712e9381d2 HDFS-13068. RBF: Add router admin option to manage safe mode. Contributed by Yiqun Lin. 2018-02-02 17:34:12 +08:00
Akira Ajisaka
01661d4f90
HDFS-13048. LowRedundancyReplicatedBlocks metric can be negative 
(cherry picked from commit 4aef8bd2efd68bf96c077ddda1538dcd5691b437)
 2018-02-02 14:35:11 +09:00
Xiao Chen
abdf8abf1a HADOOP-15197. Remove tomcat from the Hadoop-auth test bundle. 
(cherry picked from commit 09dd709d6e4022cd68187c4da1483d98ffc7e15a)
 2018-02-01 15:35:27 -08:00
Hanisha Koneru
992d7b5739 HDFS-13062. Provide support for JN to use separate journal disk per namespace. Contributed by Bharat Viswanadham. 
(cherry picked from commit dd50f53997239bf9078481cf46592ca3e41520b5)
 2018-02-01 12:32:47 -08:00
Xiaoyu Yao
784b454122 HDFS-12997. Move logging to slf4j in BlockPoolSliceStorage and Storage. Contributed by Ajay Kumar. 
(cherry picked from commit b3ae11d59790bb08b81848e9f944db7d3afbbd8a)
 2018-02-01 12:21:29 -08:00
Inigo Goiri
7491aaeb8f HDFS-13043. RBF: Expose the state of the Routers in the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 6ca7204cebf4b7060696a07b1777723a6dfa7d9d)
 2018-02-01 10:37:56 -08:00
Jason Lowe
84297f69d8 Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 874bdbc4013b72e40b64fb874e83d3d14045a93d.
 2018-02-01 12:17:23 -06:00
Xiao Chen
4010262ae0 HDFS-12897. getErasureCodingPolicy should handle .snapshot dir better. Contributed by LiXin Ge. 
(cherry picked from commit ae2177d296a322d13708b85aaa8a971b8dcce128)
 2018-01-31 22:50:52 -08:00
Xiaoyu Yao
e260a0af24 HDFS-13060. Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver. Contributed by Ajay Kumar. 
(cherry picked from commit b3ac6f7ce3f5e014af7ad6bf882431a97716e020)
 2018-01-31 22:40:13 -08:00
Eric Payne
1d53c9d3a9 MAPREDUCE-7033: Map outputs implicitly rely on permissive umask for shuffle. Contributed by Jason Lowe 
(cherry picked from commit 5a725bb886eb0b9182840c4a91466c361ecc11e0)
 2018-01-31 16:01:46 -06:00
Hanisha Koneru
b71d775f57 HDFS-13092. Reduce verbosity for ThrottledAsyncChecker#schedule. Contributed by Mukul Kumar Singh. 
(cherry picked from commit 3ce2190b581526ad2d49e8c3a47be1547037310c)
 2018-01-31 11:07:39 -08:00
Xiaoyu Yao
02790eb633 HDFS-13061. SaslDataTransferClient#checkTrustAndSend should not trust a partially trusted channel. Contributed by Ajay Kumar. 
(cherry picked from commit f4bcae10d306ef2b0083e1de17253ec732b5b5c1)
 2018-01-31 10:47:46 -08:00
Jason Lowe
874bdbc401 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 
(cherry picked from commit 12eaae383ad06de8f9959241b2451dec82cf9ceb)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
 2018-01-31 11:45:57 -06:00
Inigo Goiri
961dcb693d HDFS-13044. RBF: Add a safe mode for the Router. Contributed by Inigo Goiri. 2018-01-31 09:08:47 -08:00
Inigo Goiri
0ce4851ee2 HDFS-13083. RBF: Fix doc error setting up client. Contributed by tartarus. 
(cherry picked from commit 5206b2c7ca479dd53f614d75bab594043a9866e1)
 2018-01-30 18:27:51 -08:00
Weiwei Yang
7f62d2a473 HDFS-12528. Add an option to not disable short-circuit reads on failures. Contributed by Xiao Chen. 2018-01-31 07:35:54 +08:00
Sammi Chen
91184299c5 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302028ed3921d1016ac2fa5754f06ebf0)
 2018-01-30 15:21:27 +08:00
Kihwal Lee
673200ac1e HDFS-12574. Add CryptoInputStream to WebHdfsFileSystem read call. Contributed by Rushabh S Shah 
(cherry picked from commit fde95d463c3123b315b3d07cb5b7b7dc19f7cb73)
 2018-01-29 17:25:30 -06:00
Steve Loughran
95a96b13e2 HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.
 2018-01-29 09:46:53 -08:00
Sunil G
0463ede09d YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 
(cherry picked from commit e9c72d04beddfe0252d2e81123a9fe66bdf04078)
 2018-01-29 20:46:32 +05:30
Jinjiang Ling
1d7e9b9ad8
YARN-7698. A misleading variable's name in ApplicationAttemptEventDispatcher 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3400d0c535aeb151c3f283cc41111b15d66990e5)
 2018-01-30 00:12:40 +09:00
Xiao Chen
75be15cf02 HDFS-12974. Exception message is not printed when creating an encryption zone fails with AuthorizationException. Contributed by fang zhenyi. 
(cherry picked from commit b63dcd583f0b98e785831004f41bd7c7de8b3c18)
 2018-01-28 22:24:00 -08:00
Xiao Chen
fd5081bd24 HDFS-13065. TestErasureCodingMultipleRacks#testSkewedRack3 is failing. Contributed by Gabor Bota. 
(cherry picked from commit 6bc2f7f4b4b8d4c36e92764d4c975c17f9fdd63b)
 2018-01-28 22:12:25 -08:00
Jian He
3316779364 YARN-7765. Fixed an issue that kerberos tgt not found when NM posting timeline events. Contributed by Rohith Sharma K S 2018-01-27 22:20:49 -08:00
Steve Loughran
ec4f5f0f0d HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common 
Contributed by Genmao Yu
 2018-01-26 15:23:08 -08:00
Jason Lowe
d7acc45ce4 MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko 
(cherry picked from commit 6eef3d7f1a1e5e3f27fb3bf7596663640d786181)
 2018-01-26 15:37:16 -06:00
Arpit Agarwal
480b69c03c HDFS-13054. Handling PathIsNotEmptyDirectoryException in DFSClient delete call. Contributed by Nanda kumar. 2018-01-26 11:45:23 -08:00
Chris Douglas
7f35487788 MapFile.fix creates a wrong index file in case of block-compressed data file. Contributed by Grigori Rybkine 
(cherry picked from commit 91db424c4360d7556660e8c57ac9a266e6688e01)
 2018-01-26 09:15:03 -08:00
Haibo Chen
e6c66baba9 MAPREDUCE-7041. MR should not try to clean up at first job attempt. (Gergo Repas via Haibo Chen) 
(cherry picked from commit cc10852252c2d69294eabc68bd032cc630a53b18)
 2018-01-25 17:14:25 -08:00
Jason Lowe
217f408aaa YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit ff8378eb1b960c72d18a984c7e5d145b407ca11a)
 2018-01-25 17:51:51 -06:00
Wei Yan
c5c042135d HDFS-13049. RBF: Inconsistent Router OPTS config in branch-2 and branch-3. Contributed by Wei Yan. 
(cherry picked from commit 0c139d5bcfbcd62fc69111ee6204926c57d57bf1)
 2018-01-25 12:52:45 -08:00
Sunil G
88bacad50f YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 
(cherry picked from commit 82cc6f6968d2897ae006c5ef29eba74023f3483e)
 2018-01-25 14:33:42 +05:30