Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,094 Commits 326 Branches 371 Tags
Commit Graph

12094 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jing Zhao
e327325a37 HDFS-7964. Add support for async edit logging. Contributed by Daryn Sharp. 
(cherry picked from commit 2151716832ad14932dd65b1a4e47e64d8d6cd767)
(cherry picked from commit 3cb7ae11a839c01b8be629774874c1873f51b747)
 2016-10-31 16:24:35 -07:00
Mingliang Liu
fece3c881b HDFS-11030. TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing). Contributed by Mingliang Liu 
(cherry picked from commit 6bb12ecc3f63afb2070296c2db009c6f9a2d884b)
 2016-10-31 16:06:41 -07:00
Steve Loughran
a83f10c90d HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:53:22 +00:00
Arpit Agarwal
f0a6d17e3b HDFS-11074. Remove unused method FsDatasetSpi#getFinalizedBlocksOnPersistentStorage. Contributed by Hanisha Koneru. 2016-10-31 12:40:03 -07:00
Naganarasimha
b8ae7a8ee4 YARN-4498. Application level node labels stats to be available in REST. Contributed by Bibin A Chundatt. 
(cherry picked from commit e4023f0d8177ac37a75685b55c11991ca4ea2c23)
 2016-10-31 19:09:53 +05:30
Akira Ajisaka
757b38ea43 MAPREDUCE-6799. Document mapreduce.jobhistory.webapp.https.address in mapred-default.xml. Contributed by Yiqun Lin. 
(cherry picked from commit f646fe34be704a22cbea322cce2826f8f201bd34)
(cherry picked from commit ef13a5ef9d7859cbb720ee46ebed8172358ad0d5)
 2016-10-31 15:19:37 +09:00
Varun Saxena
367cb029d4 YARN-5773. RM recovery too slow due to LeafQueue#activateApplications (Bibin A Chundatt via Varun Saxena) 2016-10-30 11:50:54 +05:30
Wei-Chiu Chuang
9b6d27793d HDFS-11061. Update dfs -count -t command line help and documentation. Contributed by Yiqun Lin. 
(cherry picked from commit 112f04eb1b771e410ffd232a46e0ee2bc7b99d06)
(cherry picked from commit 9ba283c27d73962a894b33a89f17d3ec697bb388)
 2016-10-28 15:37:53 -07:00
Mingliang Liu
616c1194fe HDFS-11047. Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode. Contributed by Xiaobing Zhou 
(cherry picked from commit 2848a27700e76fc2c2d3ab87adba54ca8e93ba39)
 2016-10-28 11:18:26 -07:00
Mingliang Liu
a8a99f3423 HDFS-11048. Audit Log should escape control characters. Contributed by Eric Badger 
(cherry picked from commit 8a9388e5f6d622152798aaaa256064919e4fec37)
 2016-10-28 11:01:09 -07:00
Jason Lowe
93165c1ade YARN-4963. capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable. Contributed by Nathan Roberts 2016-10-28 17:44:56 +00:00
Jason Lowe
9cc7ab4e96 YARN-5767. Fix the order that resources are cleaned up from the local Public/Private caches. Contributed by Chris Trezzo 
(cherry picked from commit 1b79c417dca17bcd2e031864bc6ca07254c61b47)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
 2016-10-28 16:11:59 +00:00
Jason Lowe
83bb428e6c YARN-5027. NM should clean up app log dirs after NM restart. Contributed by sandflee 
(cherry picked from commit 7146359bfd436a76585fb1f3ea93716795308cec)
 2016-10-28 15:50:22 +00:00
Jason Lowe
558e53b10b HADOOP-13770. Shell.checkIsBashSupported swallowed an interrupted exception. Contributed by Wei-Chiu Chuang 
(cherry picked from commit c017171da00a6cd71a2901c84a0298ce14a49e23)
 2016-10-28 15:06:57 +00:00
Jason Lowe
0a405c4f71 MAPREDUCE-6724. Single shuffle to memory must not exceed Integer#MAX_VALUE. (Haibo Chen via gera) 
(cherry picked from commit 6890d5b472320fa7592ed1b08b623c55a27089c6)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java
 2016-10-28 14:40:28 +00:00
Naganarasimha
8a664eba7d MAPREDUCE-2631. Potential resource leaks in BinaryProtocol$TeeOutputStream.java. Contributed by Sunil G. 
(cherry picked from commit 28660f51af161a9fa301523d96a6f8ae4ebd6edd)
 2016-10-28 06:04:03 +05:30
Erik Krogen
2ec90a1a4b HDFS-9500. Fix software version counts for DataNodes during rolling upgrade. Contributed by Erik Krogen. 
(cherry picked from commit f3ac1f41b8fa82a0ac87a207d7afa2061d90a9bd)
 2016-10-27 16:04:58 -07:00
Carlo Curino
b32f14d806 YARN-3432. Cluster metrics have wrong Total Memory when there is reserved memory on CS. (Brahma Reddy Battula via curino) 
(cherry picked from commit 892a8348fceb42069ea9877251c413fe33415e16)
(cherry picked from commit 58ac40b55296834a8e3f3375caddc03bee901e9a)
 2016-10-27 15:23:04 -07:00
Subru Krishnan
55ba220721 YARN-3568. TestAMRMTokens should use some random port. (Takashi Ohnishi via Subru). 
(cherry picked from commit 79ae78dcbec183ab53b26de408b4517e5a151878)
(cherry picked from commit 4274600b9527706259ec5df62a13a5a66adc3ff2)
 2016-10-27 15:19:01 -07:00
Wei-Chiu Chuang
bc2f429c0e HDFS-11055. Update default-log4j.properties for httpfs to imporve test logging. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 31ff42b51037632ec871f29efc0fa894e1b738d0)
(cherry picked from commit 2cf3138d72a1392d13f6225b4a9fca4fe7ced132)
 2016-10-27 13:47:26 -07:00
Jason Lowe
d21dc8aa6c YARN-4831. Recovered containers will be killed after NM stateful restart. Contributed by Siqi Li 
(cherry picked from commit 7e3c327d316b33d6a09bfd4e65e7e5384943bb1d)
 2016-10-27 20:45:18 +00:00
Kihwal Lee
40dbf2a18d HDFS-11069. Tighten the authorization of datanode RPC. Contributed by Kihwal Lee 
(cherry picked from commit ae48c496dce8d0eae4571fc64e6850d602bae688)
 2016-10-27 14:18:48 -05:00
Kihwal Lee
5b0d32b4aa Revert "HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu" 
This reverts commit d6dfc76dfee4489b2a55cfb8562970c3212b15b5.
 2016-10-27 14:10:52 -05:00
Brahma Reddy Battula
d6dfc76dfe HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu 
(cherry picked from commit ac35ee9393e0afce9fede1d2052e7bf4032312fd)
 2016-10-27 20:23:07 +05:30
Brahma Reddy Battula
26cf988c3a HADOOP-13201. Print the directory paths when ViewFs denies the rename operation on internal dirs. Contributed by Tianyin Xiu 
(cherry picked from commit 0c837db8a874079dd5db83a7eef9c4d2b9d0e9ff)
 2016-10-27 19:51:05 +05:30
Steve Loughran
5275d37495 Revert "HADOOP-13514. Upgrade maven surefire plugin to 2.19.1. Contributed by Ewan Higgs." 
This reverts commit 444312b54b890ca7b0bfe6f997007ef0ad82b00e.
 2016-10-27 15:21:29 +02:00
Steve Loughran
ca6f0a1d77 Revert "Addendum patch for HADOOP-13514 Upgrade maven surefire plugin to 2.19.1. Contributed by Akira Ajisaka." 
This reverts commit 9a98feb769dbff87d337c4297ee8d7a945fed055.
 2016-10-27 15:21:17 +02:00
Naganarasimha
27029393b7 MAPREDUCE-6541. Exclude scheduled reducer memory when calculating available mapper slots from headroom to avoid deadlock. Contributed by Varun Saxena 
(cherry picked from commit 060558c6f221ded0b014189d5b82eee4cc7b576b)
 2016-10-27 18:06:34 +05:30
Naganarasimha
3b9df97158 YARN-3848. TestNodeLabelContainerAllocation is timing out. Contributed by Varun Saxena 
(cherry picked from commit 6c8830992c734313fc95ac9988539c4d813c3581)
 2016-10-27 17:16:34 +05:30
Vinayakumar B
7279ce831d HDFS-8492. DN should notify NN when client requests a missing block (Contributed by Walter Su) 
(cherry picked from commit 1cf6ec4ad4e1f4ea71f912923b5e8627b61ef482)
(cherry picked from commit 144f7a9e0eb3084eaaff7ab0bee1f5943e16d136)
 2016-10-27 16:56:25 +05:30
Masatake Iwasaki
6d9b6fac86 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82a4510f16df49238d57c9f78ac28ae7)
 2016-10-27 15:55:35 +09:00
Mingliang Liu
a7684f6517 HDFS-10921. TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode. Contributed by Eric Badger 
(cherry picked from commit 55e1fb8e3221941321e6f5e04b334246c5f23027)
 2016-10-26 14:18:00 -07:00
Wei-Chiu Chuang
9a98feb769 Addendum patch for HADOOP-13514 Upgrade maven surefire plugin to 2.19.1. Contributed by Akira Ajisaka. 
(cherry picked from commit e48b592f8ba1d8a89587f2c4403d861f2d015a9a)
(cherry picked from commit 0c96ceaca9d66b07d4ed5c04697b4a826812918d)
 2016-10-26 14:10:26 -07:00
Chris Nauroth
4df1780991 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5bacd98d0d371447ada3b31f93ca40a4e)
(cherry picked from commit 082d69ee663d5c1a59f32c40b9fbde9996886de9)
 2016-10-26 08:57:46 -07:00
Chris Nauroth
c940c68c79 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e235026dbe4658705ca85d263d0edf14521)
(cherry picked from commit 67e01f7218e592d7b18316d65f7b22ae8b9ad7a6)
 2016-10-26 08:27:39 -07:00
Kihwal Lee
e7bd2e8d63 HDFS-11053. Unnecessary superuser check in versionRequest(). Contributed by Kihwal Lee. 
(cherry picked from commit e90af4a89ba2741aeca20ead47699587b14f0de8)
 2016-10-26 09:18:01 -05:00
Kihwal Lee
49adab66b8 HDFS-11050. Change log level to 'warn' when ssl initialization fails and defaults to DEFAULT_TIMEOUT_CONN_CONFIGURATOR. Contributed by Kuhu Shukla. 
(cherry picked from commit ce6bbfb23c9aafaf1aaeaeceba88286d4270b316)
 2016-10-26 08:11:51 -05:00
Arpit Agarwal
df83d86da7 HDFS-11054. Suppress verbose log message in BlockPlacementPolicyDefault. Contributed by Chen Liang. 2016-10-25 19:51:53 -07:00
Xiaoyu Yao
e0fc749a46 HDFS-11011. Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'. Contributed by Xiaobing Zhou. 
(cherry picked from commit 85f6fec0370b3cd94d6c2f19920c0b6d33f127c6)
(cherry picked from commit c2ba4b650168be4fad7ffdb546b8a0d985268393)
 2016-10-25 11:22:07 -07:00
Zhe Zhang
262518fa5b HDFS-11015. Enforce timeout in balancer. Contributed by Kihwal Lee. 
(cherry picked from commit f6367c5f44a88cb5eb7edffb015b10b657504a61)
(cherry picked from commit ff806cbfc7f5d1bdccd51a1802b6d69f4777219e)
 2016-10-25 10:59:03 -07:00
Xiao Chen
a8aabf9be6 HADOOP-13669. Addendum patch 2 for KMS Server should log exceptions before throwing. 
(cherry picked from commit 09ef97dccbd4c11aaeedf7503d414f999435ad91)
 2016-10-25 09:59:48 -07:00
Chris Nauroth
3d5f41544a HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c078ff51cdb6bd1273e6f91f43311cb)
(cherry picked from commit 05d772e297fa3738b3fdddfa666bd6e23688f2b6)
 2016-10-25 09:03:19 -07:00
Akira Ajisaka
444312b54b HADOOP-13514. Upgrade maven surefire plugin to 2.19.1. Contributed by Ewan Higgs. 
(cherry picked from commit dbd205762ef2cba903b9bd9335bb9a5964d51f74)
(cherry picked from commit 6bb23a14b6f20973eded529f56c9a878a29035ff)
 2016-10-25 15:58:57 +09:00
Chris Nauroth
02f0472d1e HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 
(cherry picked from commit 3372e940303149d6258e0b72c54d72f080f0daa2)
(cherry picked from commit 5c2f67bdae208ea14c518af58e551d563b2bf8cd)
 2016-10-24 21:54:25 -07:00
Chris Nauroth
78970e0db3 HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa6722cbf7a4ec3d6b9c44b034da9aa351)
(cherry picked from commit 5b7cbb5a3c1877e51f63f6d6dfb201afa55dc4da)
 2016-10-24 21:23:00 -07:00
Kihwal Lee
c82910203c HDFS-11042. Add missing cleanupSSLConfig() call for tests that use setupSSLConfig(). Contributed by Kuhu Shukla. 
(cherry picked from commit 086577c67a458b466acc3595f5c74beffbcaab50)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/TestHttpFSFWithSWebhdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
 2016-10-24 09:00:28 -05:00
Jian He
dab611e605 Revert YARN-4126. RM should not issue delegation tokens in unsecure mode. 
(cherry picked from commit bb8214cab8a239385c2b06ad689d87b82abf6964)
 2016-10-23 16:56:12 -07:00
Xiaoyu Yao
7d826631ea HADOOP-13749. KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used. Contributed by Xiaoyu Yao. 
(cherry picked from commit d0a347984da175948b553a675dc357491df2fd0f)
 2016-10-23 10:59:10 -07:00
Xiaoyu Yao
751568e1d3 Revert "HDFS-10757. KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used. Contributed by Xiaoyu Yao." 
This reverts commit b82364759ca8600a43741e35c6781bec8249c547.
 2016-10-23 08:30:20 -07:00
Benoy Antony
653ceab7d3 HADOOP-12082 Support multiple authentication schemes via AuthenticationFilter. Contributed by Hrishikesh Gadre. 2016-10-21 19:43:22 -07:00