Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,166 Commits 326 Branches 371 Tags
Commit Graph

11166 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Chris Nauroth
5b907a17ed HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 
(cherry picked from commit 500e946729e1b85fdc0ce8475d8ce118ba886f46)
(cherry picked from commit e7b56cf93c7ac269e40b556875f16d0c5ad2f819)
 2016-05-20 13:02:06 -07:00
Chris Nauroth
877af3ec7e HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 
(cherry picked from commit bcde1562d25c4f5595f4e3436dc3630315b1ceed)
(cherry picked from commit d38f2090fa998b716a5d7a8d2314927d6eb30e41)
 2016-05-20 12:50:23 -07:00
Jian He
dbde956bf5 YARN-5020. Fix Documentation for Yarn Capacity Scheduler on Resource Calculator. Contributed by Takashi Ohnishi 
(cherry picked from commit d364ceac85622e99133b3eb3becef0c8188e6f89)
(cherry picked from commit 6e0522ee1b46b262d031eb7eddd1e7e1aa3ca736)
 2016-05-20 11:36:40 -07:00
Steve Loughran
ce17c9fd51 Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth." 
This reverts commit edb912c57123ce4cba109a41a7dc21a45da7d27c.
 2016-05-20 14:00:26 +01:00
Steve Loughran
f32b37ec89 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:51:56 +01:00
Steve Loughran
edb912c571 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:35 +01:00
Chris Nauroth
dbff8f2932 HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 
(cherry picked from commit 93258459faf56bc84121ba99d20eaef95273329e)
(cherry picked from commit 60e5397a9bb6a1d8ca6a09c2893108af77dc96c3)
 2016-05-19 22:00:42 -07:00
Rohith Sharma K S
f102c4f6c8 YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks) 2016-05-20 08:53:21 +05:30
Kihwal Lee
a81150ae9e HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 
(cherry picked from commit 204de430bf6a931e5688789c7a783c9c568a98c2)
 2016-05-19 15:54:18 -05:00
Jian He
61fa9256c5 YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong 
(cherry picked from commit feb90ffcca536e7deac50976b8a8774450fe089f)
 2016-05-19 13:02:24 -07:00
Allen Wittenauer
27d6538b88 HADOOP-13177. Native tests fail on OS X, because DYLD_LIBRARY_PATH is not defined to include libhadoop.dylib (Chris Nauroth via aw) 
(cherry picked from commit c7159f0240be0c555edf01d9a580987b0997125d)
(cherry picked from commit f7ffa1a5344517b4bb70cb329b29d25c9d63a033)
 2016-05-19 09:59:02 -07:00
Junping Du
c8843cac03 YARN-5100. The YarnApplicationState is always running in ATS even application is finished. Contributed by Xuan Gong. 
(cherry picked from commit 141873ca7d68619e3458712b17aed57fec8fd7c8)
(cherry picked from commit 416274b53eefb76cf9d3dc2992d3e22b304c321e)
 2016-05-19 09:54:03 -07:00
Steve Loughran
7a001ae19b HADOOP-12767. Update apache httpclient version to 4.5.2; httpcore to 4.4.4. Artem Aliev via stevel. 2016-05-19 14:47:46 +01:00
Rohith Sharma K S
726c1f14b8 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999). Contributed by Mohammad Shahid Khan and Varun Saxena 2016-05-19 10:50:32 +05:30
Andrew Wang
64e75c34a3 HADOOP-13157. Follow-on improvements to hadoop credential commands. Contributed by Mike Yoder. 2016-05-18 14:57:17 -07:00
Naganarasimha
0fb6a05577 YARN-4925. ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression. Contributed by Bibin A Chundatt 
(cherry picked from commit f04c81c9ce93512bc714531a7731debbe6b794ce)
 2016-05-19 01:40:06 +08:00
Chris Nauroth
4ce5a58ca9 HADOOP-13138. Unable to append to a SequenceFile with Compression.NONE. Contributed by Vinayakumar B. 
(cherry picked from commit 0a527196dcb35a699e12121165bc3738766b569e)
(cherry picked from commit 42f64b3bcc333fa416a8311b84c5b8112043eefd)
 2016-05-18 09:32:24 -07:00
Arpit Agarwal
01a0730996 HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:25:33 -07:00
Colin Patrick Mccabe
7bbf42b82f HADOOP-13140. FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes (Mingliang Liu via cmccabe) 
(cherry picked from commit f2c1d9181f80e53890b50309b15c9ea37cb24987)
(cherry picked from commit feabfe9cda127a4e0314a20d1f4d745e319d9eaa)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
 2016-05-18 09:20:31 -07:00
Xiaoyu Yao
ccb1cade5b HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 18:00:18 -07:00
Yongjun Zhang
2bcf1eb36a HDFS-10381, DataStreamer DataNode exclusion log message should be warning. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit 16c07cc68a3e0a06f57b7f4c7207cc8e5dce211f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java

(cherry picked from commit 456fe08a6dcf1168f478a1bdd9f2dc3b4ff59810)
 2016-05-17 16:00:49 -07:00
Jing Zhao
8755905e30 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015c962eac1a35fa5df39356e8b84731c)
(cherry picked from commit dd7f5a3b88cc37945bfc96ca91ed5f39d44ee394)
 2016-05-17 15:55:31 -07:00
Xiaoyu Yao
744d5edc87 HADOOP-13159. Fix potential NPE in Metrics2 source for DecayRpcScheduler. Contributed by Xiaoyu Yao. 
(cherry picked from commit 94784848456a92a6502f3a3c0074e44fba4b19c9)
 2016-05-17 15:44:28 -07:00
Kihwal Lee
01de843a31 Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu." 
This reverts commit cb5a1edd1cfebec389110a192aa0e5359abeff0e.
 2016-05-17 15:21:41 -05:00
Kihwal Lee
cb5a1edd1c HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 
(cherry picked from commit 0942954e8ab6bef636c2648d0d5a0227a50f799c)
 2016-05-17 15:11:40 -05:00
Jason Lowe
c983115e4c MAPREDUCE-6698. Increase timeout on TestUnnecessaryBlockingOnHistoryFileInfo.testTwoThreadsQueryingDifferentJobOfSameUser. Contributed by Haibo Chen 
(cherry picked from commit 34fddd1e912dcd9f898e500b8df11295ee49c0d8)
 2016-05-17 15:04:33 +00:00
Kihwal Lee
362a778bf5 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4a5819dae2b0ca8f8b6d94ef464882d079d86593)
 2016-05-17 09:00:59 -05:00
Steve Loughran
42d9beaa1d HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:18:51 +01:00
Steve Loughran
b44ecfff12 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:02:22 +01:00
Akira Ajisaka
fc221ecf3d MAPREDUCE-6693. ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit. Contributed by Ajith S. 
(cherry picked from commit b3579305268bffc44a8041c5b75f15f01ebb9ce4)
(cherry picked from commit 19db965c2776e86095ecd31d7e582aca07d1e789)
 2016-05-17 20:23:00 +09:00
Akira Ajisaka
388b2010ae HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 
(cherry picked from commit 9fe5828f05accc6746cb08a43916f7557dac533a)
(cherry picked from commit 7f03c1c90e3816b3957dbf121c6d7bc3ce07cabd)
 2016-05-17 15:56:36 +09:00
Eric Payne
a552932b38 YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger. 
(cherry picked from commit 1217c8f6b49e3afd8ca008ffddcf6615f0accfc5)
 2016-05-16 20:44:41 +00:00
Chris Nauroth
28a2f28e42 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d59e69e5dd64802fa9025dad1b315be5)
(cherry picked from commit bb41547fb42b9dd2b80ec2a7286db6d35fd251b5)
 2016-05-16 11:53:33 -07:00
Jason Lowe
70faa87ccf YARN-4325. Nodemanager log handlers fail to send finished/failed events in some cases. Contributed by Junping Du 
(cherry picked from commit 81effb7dcde2b31423438d6f1b8b8204d4ca05b3)
 2016-05-16 15:43:42 +00:00
Andrew Wang
29ed65ff9b HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 
(cherry picked from commit 45788204ae2ac82ccb3b4fe2fd22aead1dd79f0d)
(cherry picked from commit 413f3dabbd398b7d837893963556d9e6dd9e10e2)
 2016-05-15 22:15:18 -07:00
Xuan
4eeec4e35b YARN-5080. Cannot obtain logs using YARN CLI -am for either KILLED or 
RUNNING AM. Contributed by Xuan Gong

(cherry picked from commit 7be53b65f95dae89c760cde354b42f15f5ee3b5f)
(cherry picked from commit 3fe4107909393dea91adac9fda16c45814e01648)
 2016-05-14 12:22:33 -07:00
Li Lu
6b1af6ce68 HADOOP-13083. The number of javadocs warnings is limited to 100. (Gergely Novák via gtcarrera9) 
(cherry picked from commit 3fa1380c221b9d659fb82c42284505ef19da38d0)
 2016-05-13 13:46:28 -07:00
Sangjin Lee
29699d8e7e HADOOP-12971. FileSystemShell doc should explain relative path (John Zhuge via sale) 
(cherry picked from commit 1f2794b4faf119a1ab66c8184ac84cec710d52a0)
 2016-05-13 10:03:06 -07:00
Jason Lowe
ff8caccc87 MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 9227dfc25f373a99cb66ad7d6bacef8dcf336f77)
 2016-05-13 14:36:13 +00:00
Steve Loughran
a84850b4ca HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:46:15 +01:00
Andrew Wang
b5fd6dd4a8 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 
(cherry picked from commit fa440a39b7883183b9cf43c056908ca7bf68e6a1)
 2016-05-12 23:58:26 -07:00
Andrew Wang
5da54eb3d4 HADOOP-11180. Change log message "token.Token: Cannot find class for token kind kms-dt" to debug. Contributed by Yi Liu. 
(cherry picked from commit 3bfd29949fef6d8fb5df4c6398e77cde320d8c2d)
(cherry picked from commit d6f6b99c83b0945c7d125a57650041d2e707ba0e)
 2016-05-12 23:49:00 -07:00
Steve Loughran
8d74a58042 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:18 +01:00
Steve Loughran
9b8f7a8fb2 MAPREDUCE-6639 Process hangs in LocatedFileStatusFetcher if FileSystem.get throws. Ryan Blue via stevel 2016-05-12 18:59:56 +01:00
Steve Loughran
9cbb942465 HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:36:10 +01:00
Steve Loughran
af5e5d6d47 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:56:48 +01:00
Karthik Kambatla
a9b0041abc YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha) 
(cherry picked from commit 4b4e4c6ba83bc5c41d7bb69bb2483bcfe894a260)
(cherry picked from commit f96b2b6fac84f565d4d0cf78c927f0b116364cfc)
 2016-05-11 17:40:57 -07:00
Colin Patrick Mccabe
5f04795aed HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 
(cherry picked from commit 687233f20d24c29041929dd0a99d963cec54b6df)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java

(cherry picked from commit 15bfcde23832189accce8293a053c8f0ef1aeaae)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2016-05-11 14:24:51 -07:00
Varun Vasudev
4d275d0a25 HADOOP-12749. Create a threadpoolexecutor that overrides afterExecute to log uncaught exceptions/errors. Contributed by Sidharta Seethana. 
(cherry picked from commit d37eb828ffa09d55936964f555ea351b946d286e)
(cherry picked from commit 2451a0a8e08f122061b2defd0ba5b832f8689e53)
 2016-05-11 14:12:18 -07:00
Larry McCay
ee2c2afdc3 HADOOP-12942. hadoop credential commands non-obviously use password of "none" (Mike Yoder via lmccay) 2016-05-11 14:39:13 -04:00