Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,500 Commits 326 Branches 371 Tags
Commit Graph

11500 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mingliang Liu
9d304822f6 HADOOP-13527. Add Spark to CallerContext LimitedPrivate scope. (Contributed by Weiqing Yang) 
(cherry picked from commit 115ecb52a86b49aad3d058a6b4c1c7926b8b0a40)
 2016-08-21 09:42:06 -07:00
Jing Zhao
8628ad0790 HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by Arpit Agarwal. 
(cherry picked from commit 0faee62a0c8c1b8fd83227babfd00fbc2b26bddf)
(cherry picked from commit fff0418458b23c3096384082f8f387b451efad09)
 2016-08-19 22:27:24 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
b2efb2040c HADOOP-13428. Fixed hadoop-common build files to generate jdiff by working around a jdiff-bug. Contributed by Wangda Tan. 
(cherry picked from commit 99603e902244f17b04cfd55122f47355d070b588)
 2016-08-19 19:11:04 -07:00
Wangda Tan
2e9d84d24c YARN-4685. Disable AM blacklisting by default to mitigate situations that application get hanged. (Rohith Sharma K S via wangda) 
(cherry picked from commit 2da32a6ef9edebd86ca9672d10ce35b5a46818cc)
(cherry picked from commit 3e511f39c220ef65708125a37e630c495c450ad1)
 2016-08-19 16:55:40 -07:00
Wangda Tan
9bf420b3bc YARN-4837. User facing aspects of 'AM blacklisting' feature need fixing. (vinodkv via wangda) 2016-08-19 16:55:34 -07:00
Wangda Tan
3bf2e16f76 YARN-3388. Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit. (Nathan Roberts via wangda) 
(cherry picked from commit 444b2ea7afebf9f6c3d356154b71abfd0ea95b23)
(cherry picked from commit c7d782d2f6e61103f7a009c48b11dcd64e07c2ce)
 2016-08-19 16:54:03 -07:00
Wangda Tan
e2bd490236 MAPREDUCE-6310. Add jdiff support to MapReduce. (Li Lu/vinodkv via wangda) 
(cherry picked from commit 3d937457ee60d617d71b36bb6a29344e771670b4)
(cherry picked from commit 65822e55fd99511ab0c39f802d3ec58290445f9d)
 2016-08-19 16:49:54 -07:00
Varun Saxena
6f1a9f1b6a MAPREDUCE-6763. Shuffle server listen queue is too small (Jason Lowe via Varun Saxena) 2016-08-20 04:08:55 +05:30
Kihwal Lee
4f3df8e22a HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee) 
(cherry picked from commit ae90d4dd908cf3f9e9ff26fa8e92f028057a9ca1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
 2016-08-19 15:32:11 -05:00
Chris Nauroth
dd6c149c4a HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb996e331e40caed9ca0af966f5b3fac)
(cherry picked from commit 6dba38f08ba84109b01c8ce8d30c71354d089dbe)
 2016-08-19 12:55:57 -07:00
Jing Zhao
5d91f9fe35 HADOOP-13503. Improve SaslRpcClient failure logging. Contributed by Xiaobing Zhou. 
(cherry picked from commit 9fdd1ea72497f7a8aadbfedeba36688e0c16c52e)
 2016-08-18 15:00:17 -07:00
Mingliang Liu
ef38aa64c6 HADOOP-13512. ReloadingX509TrustManager should keep reloading in case of exception. (Contributed by Mingliang Liu) 
(cherry picked from commit 0f51eae0c085ded38216824377acf8122638c3a5)
 2016-08-18 14:22:58 -07:00
Varun Vasudev
5b49bcaee3 YARN-4307. Display blacklisted nodes for AM container in the RM web UI. Contributed by Naganarasimha G R. 
(cherry picked from commit 308d63f382e1992ea2b8ccf3130edaaa751c644d)
(cherry picked from commit 99966b1912320326fb63be1ce6d4222d736ca365)
 2016-08-18 12:33:52 -07:00
Karthik Kambatla
42a88f9c58 YARN-3446. FairScheduler headroom calculation should exclude nodes in the blacklist. (Zhihai Xu via kasha) 
(cherry picked from commit 9d04f26d4c42170ee3dab2f6fb09a94bbf72fc65)
(cherry picked from commit f0923819c35b8c499a9bcd0c6c78511f3a400fef)
 2016-08-18 12:30:39 -07:00
Chris Nauroth
1c4e039ba7 HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss. 
(cherry picked from commit ae4db2544346370404826d5b55b2678f5f92fe1f)
(cherry picked from commit 804a40720bd5cf04415831b7bb33e98d11b2046e)
 2016-08-18 10:45:20 -07:00
Steve Loughran
c09797ce9f HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:35:26 +01:00
Arpit Agarwal
1fe08c919a HDFS-10682. Replace FsDatasetImpl object lock with a separate lock object. (Contributed by Chen Liang) 2016-08-17 16:22:07 -07:00
Xiao Chen
b89d79ca1d HDFS-10549. Correctly revoke file leases when closing files. Contributed by Yiqun Lin. 
(cherry picked from commit 85aacaadb5a3f8c78b191867c0bde09b3c4b3c3c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java

(cherry picked from commit a20b943cf951cf38eb4950177c826bbcf424aade)
 2016-08-17 16:00:33 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
d65024eddc Revert "HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla." 
Reverting this on branch-2.* as it's an incompatible change.

This reverts commit 83f7f62be379045ad6933689b21b76c7086f919d.

(cherry picked from commit 2486c4c63a35fcef7338ea63f0d8aafa778cd05d)
 2016-08-17 13:34:35 -07:00
Mingliang Liu
00eb790622 HADOOP-13470. GenericTestUtils$LogCapturer is flaky. (Contributed by Mingliang Liu) 2016-08-16 17:41:51 -07:00
Chris Nauroth
ac779f2554 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 
(cherry picked from commit 3808876c7397ea68906bc5cc18fdf690c9c42131)
(cherry picked from commit 555a57adf2fa5f2d50c1141da907ae4ca032db2c)
 2016-08-16 17:11:12 -07:00
Mingliang Liu
b6d92c61fe Revert "HADOOP-13470. GenericTestUtils$LogCapturer is flaky. (Contributed by Mingliang Liu)" 
This reverts commit 657064e59397feb92cec95094ac34c85e3040c50.
 2016-08-16 16:27:36 -07:00
Andrew Wang
8b6eeb2595 HADOOP-13494. ReconfigurableBase can log sensitive information. Contributed by Sean Mackrory. 
(cherry picked from commit 4b689e7a758a55cec2ca8398727feefc8ac21bfd)
(cherry picked from commit c91e6f28283d1f8ad4504a2338a01e284ced6b24)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
 2016-08-16 15:14:06 -07:00
Kihwal Lee
b76f6e5de3 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 
(cherry picked from commit b10c936020e2616609dcb3b2126e8c34328c10ca)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 65a41696b2a22834f4592bcb49e40fffd2dbb96e)
 2016-08-16 09:39:33 -05:00
Mingliang Liu
54c9743263 HDFS-10641. TestBlockManager#testBlockReportQueueing fails intermittently. (Contributed by Daryn Sharp) 2016-08-15 20:46:48 -07:00
Mingliang Liu
657064e593 HADOOP-13470. GenericTestUtils$LogCapturer is flaky. (Contributed by Mingliang Liu) 2016-08-15 20:46:40 -07:00
Mingliang Liu
e9eaad0e67 HDFS-10747. o.a.h.hdfs.tools.DebugAdmin usage message is misleading. (Contributed by Mingliang Liu) 2016-08-15 20:46:34 -07:00
Mingliang Liu
0b934c375e HDFS-10678. Documenting NNThroughputBenchmark tool. (Contributed by Mingliang Liu) 2016-08-15 20:46:28 -07:00
Mingliang Liu
6471ec31bc HDFS-10724. Document caller context config keys. (Contributed by Mingliang Liu) 
(cherry picked from commit 4bcbef39f7ca07601092919a7f2bea531a2dfa07)
 2016-08-15 20:46:03 -07:00
Kihwal Lee
1a8280edde HDFS-10763. Open files can leak permanently due to inconsistent lease update. Contributed by Kihwal Lee. 
(cherry picked from commit 864f878d5912c82f3204f1582cfb7eb7c9f1a1da)
(cherry picked from commit e78db7d2a430983807750666fb72ebd5c97ce867)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2016-08-15 17:37:39 -05:00
Varun Saxena
6fed7b9c45 HADOOP-13333. testConf.xml ls comparators in wrong order (Vrushali C via Varun Saxena) 2016-08-16 03:14:24 +05:30
Jason Lowe
74831973cb YARN-4566. Fix test failure in TestMiniYarnClusterNodeUtilization. (Takashi Ohnishi via rohithsharmaks) 
(cherry picked from commit e0b14f26f5201a149218276469434df511697acc)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2016-08-15 19:45:45 +00:00
Junping Du
1a38dd9cee YARN-4916. TestNMProxy.tesNMProxyRPCRetry fails. Contributed by Tibor Kiss. 
(cherry picked from commit 00058167431475c6e63c80207424f1d365569e3a)
 2016-08-15 19:18:36 +00:00
Kihwal Lee
579709fc22 HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 
(cherry picked from commit 83e57e083f2cf6c0de8a46966c5492faeabd8f2a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java

(cherry picked from commit 4766a3153dd517ac832d4761c884ed88a83a6c09)
 2016-08-15 13:13:49 -05:00
Varun Saxena
ca139a3f87 YARN-5476. Non existent application reported as ACCEPTED by YarnClientImpl (Junping Du via Varun Saxena) 2016-08-13 00:27:17 +05:30
Wei-Chiu Chuang
53f097a474 HDFS-10731. FSDirectory#verifyMaxDirItems does not log path name. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9019606b69bfb7019c8642b6cbcbb93645cc19e3)
(cherry picked from commit 20a8ff0a8d17c05305f712da887ad6665eb654b5)
 2016-08-12 06:11:22 -07:00
Zhe Zhang
9c1c48c0d4 HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 
(cherry picked from commit 3e8099a45a4cfd4c5c0e3dce4370514cb2c90da9)
(cherry picked from commit 00ff3d737c71e1a228ede64a6b9abd5227ef2117)
 2016-08-11 12:42:17 -07:00
Akira Ajisaka
f29d0739d7 HDFS-10336. TestBalancer failing intermittently because of not reseting UserGroupInformation completely. Contributed by Yiqun Lin. 
(cherry picked from commit 932aed64d77edcc8483a95c1ce31a4c9ae679446)
(cherry picked from commit 0529da723966e03c7dfa8201a515320c904d32a7)
 2016-08-11 12:42:17 -07:00
Zhe Zhang
b8b4ea67d8 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
(cherry picked from commit ccff6035f50769eb69701128ae61efc69e82609d)
(cherry picked from commit 7d402692b4b06af0459ea81129e93a5182291a27)
 2016-08-11 12:42:17 -07:00
Wei-Chiu Chuang
b0bc7681c3 HADOOP-13190. Mention LoadBalancingKMSClientProvider in KMS HA documentation. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit db719ef125b11b01eab3353e2dc4b48992bf88d5)
(cherry picked from commit 6d53e096b2d96b9aace46fa9f34cd93e7f0f22a6)
 2016-08-11 12:33:15 -07:00
Rohith Sharma K S
f7cc0af611 YARN-5492. TestSubmitApplicationWithRMHA is failing sporadically during precommit builds. Contributed by Vrushali C. 
(cherry picked from commit 5199db387d59f7233a0e52ac298df31e8ed8af20)
 2016-08-11 11:52:44 +05:30
Wei-Chiu Chuang
65e3657149 HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 
(cherry picked from commit f1d5a95cf31f593f362a4e9c0afa7587f6e14957)
 2016-08-10 22:30:06 -07:00
Xiaoyu Yao
7a1a3f7754 HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit ec289bbeceff064ad24e189db20a3e0a296822c1)
 2016-08-10 15:56:56 -07:00
Xiaoyu Yao
8808df09c6 Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca74c6e6c43ecfc8770228c518620c9cb.

(cherry picked from commit fdafd0cb4f1626ee41539f4b80711c4e78e3e16c)
 2016-08-10 15:56:43 -07:00
Xiaoyu Yao
c0e331935d Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit 9f816d7ca74c6e6c43ecfc8770228c518620c9cb)
 2016-08-10 15:19:10 -07:00
Xiaoyu Yao
cc49133281 HADOOP-13285. DecayRpcScheduler MXBean should only report decayed CallVolumeSummary. Contributed by Xiaoyu Yao. 
(cherry picked from commit 0761379fe45898c44c8f161834c298ef932e4d8c)
 2016-08-10 15:19:04 -07:00
Jason Lowe
69217ba813 YARN-5382. RM does not audit log kill request for active applications. Contributed by Vrushali C 
(cherry picked from commit 5c95bb315ba605b3bed77966a99a63854234e2c5)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAuditLogger.java
 2016-08-10 19:15:12 +00:00
Wei-Chiu Chuang
753edc4931 HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 
(cherry picked from commit 8efd4959f3fd48fe281aa26a46668071461dee8b)
 2016-08-10 11:39:12 -07:00
Yuanbo Liu
d46f5ca7e2 HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 
(cherry picked from commit 10e84c6a6e831fe2bea061fb21bd0dfe32bc9953)
 2016-08-10 11:12:21 -07:00
Jason Lowe
e1845faeda YARN-5483. Optimize RMAppAttempt#pullJustFinishedContainers. Contributed by sandflee 
(cherry picked from commit 3d401206cf1eca5360f7f53165732a73d644a1fb)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
 2016-08-10 18:09:11 +00:00