Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,624 Commits 326 Branches 371 Tags
Commit Graph

17624 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Xiao Chen
1fbd419957 HDFS-13164. File not closed if streamer fail with DSQuotaExceededException. 
(cherry picked from commit 51088d323359587dca7831f74c9d065c2fccc60d)
 2018-02-23 14:00:25 -08:00
Kihwal Lee
107efcdc5d HADOOP-9747. Reduce unnecessary UGI synchronization. Contributed by Daryn Sharp. 
(cherry picked from commit 1f0952bfc734f82339d555e764e454d5594bb84f)
 2018-02-23 13:14:44 -06:00
Xiaoyu Yao
ec1b445c62 HDFS-13136. Avoid taking FSN lock while doing group member lookup for FSD permission check. Contributed by Xiaoyu Yao. 
(cherry picked from commit e978e1d9064fc794ea83217d0af2315d4167211e)
 2018-02-22 20:18:02 -08:00
Steve Loughran
c2bbe22c5a HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-21 11:11:28 +00:00
Steve Loughran
e011650090 HADOOP-15247. Move commons-net up to 3.6. 
Contributed by Steve Loughran.

(cherry picked from commit 004b722372de67635a24e71b264b3b604df4b693)
 2018-02-21 10:42:07 +00:00
Sunil G
09bb378c10 YARN-7947. Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps. Contributed by Eric Payne. 
(cherry picked from commit bdd2a184d78379d99c802a43ebec7d2cef0bbaf7)
 2018-02-21 15:06:28 +05:30
Anu Engineer
c0d8103ba2 HDFS-13175. Add more information for checking argument in DiskBalancerVolume. 
Contributed by  Lei (Eddy) Xu.

(cherry picked from commit 121e1e1280c7b019f6d2cc3ba9eae1ead0dd8408)
 2018-02-20 19:59:31 -08:00
Yiqun Lin
7601489995 HDFS-13119. RBF: Manage unavailable clusters. Contributed by Yiqun Lin. 
(cherry picked from commit 8896d20b91520053a6bbfb680adb345cd24f4142)
 2018-02-20 13:06:26 -08:00
Jason Lowe
85c611ad7d YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 2018-02-19 14:38:49 -06:00
Rohith Sharma K S
313c38b502 YARN-7937. Fix http method name in Cluster Application Timeout Update API example request. Contributed by Charan Hebri. 
(cherry picked from commit 87bdde69431c19a22d79a767071f6ea47e1ceb3d)
 2018-02-18 14:06:29 +05:30
Jason Lowe
f173e9cc04 HADOOP-15206. BZip2 drops and duplicates records when input split size is small. Contributed by Aki Tanaka 
(cherry picked from commit 0898ff42e9e5c53f2fce7ccdeb4e1cd7d0f123b3)
 2018-02-16 14:54:55 -06:00
Eric Payne
cf933e17bc MAPREDUCE-7053: Timed out tasks can fail to produce thread dump. Contributed by Jason Lowe. 
(cherry picked from commit 82f029f7b50679ea477a3a898e4ee400fa394adf)
 2018-02-16 08:25:52 -06:00
Kihwal Lee
7b83fca453 HDFS-13112. Token expiration edits may cause log corruption or deadlock. Contributed by Daryn Sharp. 
(cherry picked from commit 47473952e56b0380147d42f4110ad03c2276c961)
 2018-02-15 15:35:27 -06:00
Jason Lowe
b9ce463fbd MAPREDUCE-7052. TestFixedLengthInputFormat#testFormatCompressedIn is flaky. Contributed by Peter Bacsko 
(cherry picked from commit a53d62ab26e170a0338f93e228718da52e9196e4)
 2018-02-15 15:21:05 -06:00
Kihwal Lee
11ff6327af xattr api cleanup 
(cherry picked from commit da59acd8ca9ab5b49b988ffca64e8cce91c5f741)
 2018-02-15 11:14:42 -06:00
Steve Loughran
b302f728f8 HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b79324b7f6231b879c0bfc608203b980)
 2018-02-15 16:29:30 +00:00
Steve Loughran
1888b94806 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a31307f19551f87be4e81d4fb626de1)
 2018-02-15 14:26:18 +00:00
Anu Engineer
4b4157441f HADOOP-15204. Add Configuration API for parsing storage sizes. Contributed by Anu Engineer. 
(cherry picked from commit 8f66affd6265c9e4231e18d7ca352fb3035dae9a)
 2018-02-14 13:30:33 -08:00
Eric Payne
55ded70716 Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit 36c9dda07f9d0ac14f0c979c7d519536fc9b3d28.
 2018-02-14 14:38:31 -06:00
Xiao Chen
2f3415a4d6 Revert "HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. Contributed by Ajay Kumar." 
This reverts commit 16b2cad8e8f5215325c51eb582f58640a386b06b.
 2018-02-14 10:25:15 -08:00
Eric Payne
36c9dda07f YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-14 08:58:28 -06:00
Aaron Fabbri
e6f6c0bfc2
HADOOP-15040 Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation 2018-02-13 20:20:46 -08:00
Arpit Agarwal
16b2cad8e8 HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. Contributed by Ajay Kumar. 2018-02-13 10:14:29 -08:00
Jason Lowe
bc2d4ba507 MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko 
(cherry picked from commit 87e2570a1419d3616de2de3b553108ad1a8af425)
 2018-02-12 14:49:35 -06:00
Brahma Reddy Battula
055d5e0b85 HDFS-8693. Addendum patch to execute the command using UGI. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 35c17351cab645dcc72e0d2ae1608507aa787ffb)
 2018-02-12 22:17:35 +05:30
Arpit Agarwal
41cf2b8e7c HDFS-10453. ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.. Contributed by He Xiaoqiao. 2018-02-12 07:16:48 -08:00
Yiqun Lin
38febc0ae2 HDFS-13099. RBF: Use the ZooKeeper as the default State Store. Contributed by Yiqun Lin. 
(cherry picked from commit 543f3abbee79d7ec70353f0cdda6397ee001324e)
 2018-02-09 13:59:38 +08:00
Xiaoyu Yao
49ab09f00b HDFS-13120. Snapshot diff could be corrupted after concat. Contributed by Xiaoyu Yao. 
(cherry picked from commit 8faf0b50d435039f69ea35f592856ca04d378809)
 2018-02-08 10:18:35 -08:00
Manoj Govindassamy
bef769e436 HDFS-12985. NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted. 
(cherry picked from commit 73ff09b79a5cf9932edc21c58f3a730f7379086b)

(cherry picked from commit 821729905ed8439ee5f15372faaf6b40aff7f155)
 2018-02-07 19:49:27 -08:00
Yongjun Zhang
5105b60614 HDFS-13115. In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted. Contributed by Yongjun Zhang. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java

(cherry picked from commit 60d82ac61f2afa2a61c8eeabf20e45b5a89311d0)
 2018-02-07 16:49:26 -08:00
Anu Engineer
2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bdf9fb7aa9181e866eea383f89bf171f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb35dbe30c75c648bf759e8aeb715834a)
 2018-02-07 11:16:53 -08:00
Brahma Reddy Battula
c6349c3204 HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up. Contributed by Jianfei Jiang. 
(cherry picked from commit 01bd6ab18fa48f4c7cac1497905b52e547962599)
 2018-02-07 23:27:24 +05:30
Billie Rinaldi
0ec45f73a1 Revert "YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi." 
This reverts commit ef212b8550bd607e9e166a99f76aae4558729464.
 2018-02-07 07:51:56 -08:00
Xiao Chen
49de91446f HDFS-12933. Improve logging when DFSStripedOutputStream failed to write some blocks. Contributed by chencan. 
(cherry picked from commit e5c2fdddd69ceca21d641de56904648acedc74bf)
 2018-02-06 22:57:15 -08:00
Wei-Chiu Chuang
29a3b64ec0 HDFS-11187. Optimize disk access for last partial chunk checksum of Finalized replica. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 2021f4bdce3b27c46edaad198f0007a26a8a1391)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBuilder.java
 2018-02-06 22:57:15 -08:00
Kihwal Lee
d436c403b2 HADOOP-15212. Add independent secret manager method for logging expired tokens. Contributed by Daryn Sharp. 
(cherry picked from commit 2dd960de983a30bf0d9ee957bdb09f825f9d40a3)
 2018-02-06 15:34:52 -06:00
Billie Rinaldi
615bd15072 YARN-7890. NPE during container relaunch. Contributed by Jason Lowe 
(cherry picked from commit a196ee9362a1b35e5de20ee519f7c544ab1588e1)
 2018-02-06 11:34:49 -08:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Jason Lowe
07f779483c YARN-7849. TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error. Contributed by Botong Huang 
(cherry picked from commit 60656bcecadd80e28c81bc943b44abf13d20abae)
 2018-02-05 17:17:15 -06:00
Yongjun Zhang
35247cca87 HDFS-13100. Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. Contributed by Aaron T. Myers and Yongjun Zhang. 
(cherry picked from commit 4e9a59ce16e81b4bd6fae443a997ef24d588a6e8)

(cherry picked from commit 0524fac21ef8a8e440a2fe759936e6fb6a63cbf1)
 2018-02-04 22:53:09 -08:00
Yiqun Lin
712e9381d2 HDFS-13068. RBF: Add router admin option to manage safe mode. Contributed by Yiqun Lin. 2018-02-02 17:34:12 +08:00
Akira Ajisaka
01661d4f90
HDFS-13048. LowRedundancyReplicatedBlocks metric can be negative 
(cherry picked from commit 4aef8bd2efd68bf96c077ddda1538dcd5691b437)
 2018-02-02 14:35:11 +09:00
Xiao Chen
abdf8abf1a HADOOP-15197. Remove tomcat from the Hadoop-auth test bundle. 
(cherry picked from commit 09dd709d6e4022cd68187c4da1483d98ffc7e15a)
 2018-02-01 15:35:27 -08:00
Hanisha Koneru
992d7b5739 HDFS-13062. Provide support for JN to use separate journal disk per namespace. Contributed by Bharat Viswanadham. 
(cherry picked from commit dd50f53997239bf9078481cf46592ca3e41520b5)
 2018-02-01 12:32:47 -08:00
Xiaoyu Yao
784b454122 HDFS-12997. Move logging to slf4j in BlockPoolSliceStorage and Storage. Contributed by Ajay Kumar. 
(cherry picked from commit b3ae11d59790bb08b81848e9f944db7d3afbbd8a)
 2018-02-01 12:21:29 -08:00
Inigo Goiri
7491aaeb8f HDFS-13043. RBF: Expose the state of the Routers in the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 6ca7204cebf4b7060696a07b1777723a6dfa7d9d)
 2018-02-01 10:37:56 -08:00
Jason Lowe
84297f69d8 Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 874bdbc4013b72e40b64fb874e83d3d14045a93d.
 2018-02-01 12:17:23 -06:00
Xiao Chen
4010262ae0 HDFS-12897. getErasureCodingPolicy should handle .snapshot dir better. Contributed by LiXin Ge. 
(cherry picked from commit ae2177d296a322d13708b85aaa8a971b8dcce128)
 2018-01-31 22:50:52 -08:00
Xiaoyu Yao
e260a0af24 HDFS-13060. Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver. Contributed by Ajay Kumar. 
(cherry picked from commit b3ac6f7ce3f5e014af7ad6bf882431a97716e020)
 2018-01-31 22:40:13 -08:00
Eric Payne
1d53c9d3a9 MAPREDUCE-7033: Map outputs implicitly rely on permissive umask for shuffle. Contributed by Jason Lowe 
(cherry picked from commit 5a725bb886eb0b9182840c4a91466c361ecc11e0)
 2018-01-31 16:01:46 -06:00