Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,685 Commits 326 Branches 371 Tags
Commit Graph

19685 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ayush Saxena
4af5fc86b2 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:24:41 +05:30
Wanqiang Ji
f3ac1dde5e
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183) 
(cherry picked from commit dc5470ae86b96d6f84abd7354c3c056d3627871b)
 2020-08-06 09:37:28 +09:00
Eric Badger
647be0c0f6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 23:14:27 +00:00
Mingliang Liu
1a53bc6ecf
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:13:41 -07:00
Mingliang Liu
770735d330
HADOOP-17184. Add --mvn-custom-repos parameter to yetus calls (#2193) 
(cherry picked from commit 49996f67f782dfcf2deaac428a927f0de6c71def)
 2020-08-05 10:52:54 +09:00
sguggilam
31c61f4363
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:57:52 -07:00
Eric Badger
87816e8a51 YARN-10363. TestRMAdminCLI.testHelp is failing in branch-2.10. Contributed by 
Bilwa S T.
 2020-07-31 22:39:39 +00:00
Uma Maheswara Rao G
1768618ab9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51aefd0fd3b4602adc844ab0f172835e3)
 2020-07-31 01:42:22 -07:00
Uma Maheswara Rao G
544602e3d1 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e700066394fb9f516e23537d8abb4661409cae1)
 2020-07-31 01:42:03 -07:00
Uma Maheswara Rao G
7084b273ac HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2f2ba0bd3919650ea231cedd424f77a)
 2020-07-31 01:41:45 -07:00
Uma Maheswara Rao G
ab43b7bcfb HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc9b45413febd33840b8a653e3e29440)
 2020-07-31 01:41:21 -07:00
Uma Maheswara Rao G
49a7f9ff7b HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5024aeb7fb1f8c31b8fdd0fdb88bb66)
 2020-07-31 01:40:57 -07:00
Eric E Payne
8d8366166e YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 
(cherry picked from commit e0c9653166df48a47267dbc81d124ab78267e039)
(cherry picked from commit 863689ff9a34d9712541a06757c34499b6005f12)
 2020-07-30 17:27:35 +00:00
ywheel
2dfc6bf471
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338) 
(cherry picked from commit cf4eb75608527a8a56ad9d13a3009ccc38b29c8e)
 2020-07-30 13:31:18 +09:00
Jonathan Hung
b0edec2918 YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 
(cherry picked from commit ffb920de2a60558a9863afb31b566b8c688a8263)
 2020-07-28 13:44:40 -07:00
Eric Badger
2255bcff87 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef0f0ad4210e4027f53877760fa606a5)
 2020-07-24 23:02:19 +00:00
Akira Ajisaka
416edc5a7b
HADOOP-17147. Dead link in hadoop-kms/index.md.vm. Contributed by Xieming Li. 
(cherry picked from commit d5b476615820a7fa75b41e323db5deb5c2ed3bd5)
 2020-07-23 00:41:22 +09:00
Ayush Saxena
4592af898b HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 19:05:13 +05:30
Chen Liang
1e33696bc8 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e087518196bcb77ac2479c66a0b188d8)
(cherry picked from commit 370efc6d95fde8065dd5ba742446d0e9e570a5e9)
 2020-07-20 15:24:07 -07:00
Akira Ajisaka
635067d03d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
(cherry picked from commit 0d949d375e3e7602978f094f7ef7952783ab8421)
 2020-07-20 13:37:08 -07:00
Mukund Thakur
cb3e378782 HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-07-17 10:40:01 +01:00
He Xiaoqiao
b611559755 HDFS-14498. LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-17 00:08:42 +08:00
S O'Donnell
1a11c4bc71 HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-07-16 12:43:29 +01:00
Ahmed Hussein
b7b9bd32db HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
 2020-07-15 12:19:42 -05:00
Ahmed Hussein
34a6cbb5bb HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f9ef910654574f69591fcdc621523af)
(cherry picked from commit 43a865dc07437e0f767a902c4cfe6ca0472bf57f)
 2020-07-15 11:01:34 -05:00
Erik Krogen
0de48933c5 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:33:38 -07:00
Eric Badger
4cf5c282d0 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan.

(cherry picked from commit 09f1547697d0aa51380a0351df6d77f54af074a0)
 2020-07-14 18:27:23 +00:00
Eric E Payne
ae4e6d1677 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b7543d412f4fafe631b7ace289662d28c)
 2020-07-13 21:37:37 +00:00
Eric Badger
2de0f99539 Revert "HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell." 
This reverts commit 0bfd1b5a945ec341a36678aa3c5318f1598fc716.
 2020-07-13 21:12:04 +00:00
He Xiaoqiao
0bfd1b5a94 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:41:22 +08:00
Siyao Meng
47df2245d3 HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9d59cc46882df506dcea386020b1e4d)
 2020-07-09 16:32:07 -07:00
Uma Maheswara Rao G
4e1abe61a2 HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5584ec220b3eeaf76da85d228bcbd8b)
 2020-07-09 16:29:36 -07:00
Masatake Iwasaki
53cfe60904
HADOOP-17120. Fix failure of docker image creation due to pip2 install error. (#2130) 2020-07-09 11:35:52 +09:00
Masatake Iwasaki
4fa8055aa4 YARN-10347. Fix double locking in CapacityScheduler#reinitialize in branch-3.1. 2020-07-09 08:19:15 +09:00
Shanyu Zhao
d787b9373a HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-07 12:01:23 +08:00
Ayush Saxena
1e0424f39a HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 12:27:12 +05:30
Akira Ajisaka
2ba34ead1b HADOOP-17090. Increase precommit job timeout from 5 hours to 20 hours. (#2111). Contributed by Akira Ajisaka. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-07-01 15:17:12 +05:30
Abhishek Das
5f84093cf8 HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 13:02:00 +05:30
Masatake Iwasaki
d84386ccf7
HADOOP-17097. start-build-env.sh fails in branch-3.1. (#2108) 2020-06-30 10:47:41 +09:00
Eric E Payne
b03e6fc44a YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit e6794f2fc4d763459ce13ffa8db4c064bcb076dc)
 2020-06-29 19:37:25 +00:00
Akira Ajisaka
d26dcf6686
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac3370e220995bad0ae8b59d962c8d30a7)
 2020-06-28 16:43:32 +09:00
Virajith Jalaparti
c4e0098bd6 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a37404e9defda13a5bffe5609e72466e46)
 2020-06-27 16:25:26 -07:00
Uma Maheswara Rao G
30d1d29076 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155496cf9d82e121dd1b65d0072312197)
 2020-06-27 16:09:36 -07:00
Uma Maheswara Rao G
81e62613fb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889bbeabc73c95a4c83f0cd204365ff410)
 2020-06-27 16:09:16 -07:00
Uma Maheswara Rao G
e17770dec6 HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa6015253866ccc0ccc7247ad7024c0730)
 2020-06-27 16:08:57 -07:00
Uma Maheswara Rao G
208701329c HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
 2020-06-27 16:08:38 -07:00
Ayush Saxena
b720d7770d HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c498531bf3c011f1b0ace8eddddf61f2)
 2020-06-27 16:08:10 -07:00
Abhishek Das
0e9b6b0124 HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b658b2a1f216fbedc65938d9b6ca6e9)
 2020-06-27 16:07:49 -07:00
Uma Maheswara Rao G
100c13967e HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def959fab6b8b7ffff66410bcd240feee13)
(cherry picked from commit 120ee793fc4bcbf9d1945d5e38e3ad5b2b290a0e)
 2020-06-27 16:07:28 -07:00
Ayush Saxena
f7e590aaff HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 
(cherry picked from commit cc671b16f7b0b7c1ed7b41b96171653dc43cf670)
(cherry picked from commit bee2846bee4ae676bdc14585f8a3927a9dd7df37)
 2020-06-27 16:06:52 -07:00