Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,044 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24044 Commits

Author SHA1 Message Date
Sneha Vijayarajan eed06b46eb
Hadoop-17015. ABFS: Handling Rename and Delete idempotency 
Contributed by Sneha Vijayarajan.
 2020-07-25 13:08:01 +00:00
Eric Badger 7ec692aa83 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef)
 2020-07-24 22:46:57 +00:00
bilaharith 1ae72d2438
HADOOP-17092. ABFS: Making AzureADAuthenticator.getToken() throw HttpException 
- Contributed by Bilahari T H

Change-Id: Id9576d9509faaf057bf419ccb1879ac0cef7a07b
 2020-07-22 18:26:36 +01:00
Akira Ajisaka d35f7eaeb9
HADOOP-17147. Dead link in hadoop-kms/index.md.vm. Contributed by Xieming Li. 
(cherry picked from commit d5b4766158)
 2020-07-23 00:40:31 +09:00
Ayush Saxena e3b8d4eb05 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
Chen Liang c8c40be761 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
 2020-07-20 15:22:34 -07:00
Ayush Saxena 8a58a12626 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 2020-07-20 22:26:11 +05:30
Steve Loughran 5aa9396a58
HADOOP-17107. hadoop-azure parallel tests not working on recent JDKs (#2118) 
Contributed by Steve Loughran.

Change-Id: I972264aed36f384b7ae23e214326ef7870261cf5
 2020-07-20 10:54:22 +01:00
bilaharith e01852181a
HADOOP-16682. ABFS: Removing unnecessary toString() invocations 
- Contributed by Bilahari T H

Change-Id: Id55495b44d81533d1d3654de2553c709f505f7eb
 2020-07-20 10:53:59 +01:00
Mukund Thakur 187b161c20
HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: Ic0c9c44a1b14b4bfa962cede3f782a98fdfd7513
 2020-07-16 18:15:40 +01:00
Mehakmeet Singh 0d88ed2794
HADOOP-17129. Validating storage keys in ABFS correctly (#2141) 
Contributed by Mehakmeet Singh

Change-Id: I8016ee2f9ffbc86ea867f4a3d960b134e507d099
 2020-07-16 18:11:52 +01:00
Ahmed Hussein 9e7266df6c HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 11:40:13 -05:00
Erik Krogen 67e01ed2ca HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 
(cherry picked from 317fe4584a)
 2020-07-15 08:26:38 -07:00
Ahmed Hussein 5969922305 HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
 2020-07-15 09:57:36 -05:00
Shashikant Banerjee 292e22578a HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 
(cherry picked from commit 82343790ee)
 2020-07-15 13:13:27 +01:00
Mukund Thakur 8b601ad7e6 HADOOP-17022. Tune S3AFileSystem.listFiles() API. 
Contributed by Mukund Thakur.

Change-Id: I17f5cfdcd25670ce3ddb62c13378c7e2dc06ba52
 2020-07-14 15:28:27 +01:00
Anoop Sam John cac2fc1f58 HADOOP-16998. WASB : NativeAzureFsOutputStream#close() throwing IllegalArgumentException (#2073) 
Contributed by Anoop Sam John.
 2020-07-14 14:08:46 +01:00
Eric Badger 41bcef9486 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan

(cherry picked from commit 48f90115b5)
 2020-07-13 23:12:18 +00:00
Eric E Payne 7044a007b3 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b75)
 2020-07-13 19:02:40 +00:00
jimmy-zuber-amzn 79fc58def3
HADOOP-17105. S3AFS - Do not attempt to resolve symlinks in globStatus (#2113) 
Contributed by Jimmy Zuber.

Change-Id: I2f247c2d2ab4f38214073e55f5cfbaa15aeaeb11
 2020-07-13 19:09:50 +01:00
Steve Loughran a51d72f0c6 HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. 
Contributed by Steve Loughran.

Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
 2020-07-13 13:32:04 +01:00
He Xiaoqiao e7e7a6d503 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:31:50 +08:00
Siyao Meng 358934059f HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9)
 2020-07-09 16:30:58 -07:00
Uma Maheswara Rao G f85ce2570e HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5)
 2020-07-09 16:26:04 -07:00
Brahma Reddy Battula 7b175739a9 YARN-10341. Yarn Service Container Completed event doesn't get processed. Contributed by Bilwa S T. 
(cherry picked from commit dfe60392c9)
 2020-07-09 12:36:21 +05:30
Akira Ajisaka 0aa2d7d506
YARN-10344. Sync netty versions in hadoop-yarn-csi. (#2126) 
(cherry picked from commit 10d218934c)
 2020-07-09 15:07:44 +09:00
Masatake Iwasaki 7522b44067 HADOOP-17120. Fix failure of docker image creation due to pip2 install error. (#2130) 
(cherry picked from commit 53cfe60904)
 2020-07-09 03:19:18 +00:00
Sebastian Nagel f9619b0b97
HADOOP-17117 Fix typos in hadoop-aws documentation (#2127) 
(cherry picked from commit 5b1ed2113b)
 2020-07-09 00:04:46 +09:00
Shanyu Zhao 10c9df1d0a HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-07 11:44:42 +08:00
Madhusoodan Pataki 0789ae5b78 HADOOP-17081. MetricsSystem doesn't start the sink adapters on restart (#2089) 
Contributed by Madhusoodan P
 2020-07-06 16:26:48 +01:00
Akira Ajisaka 20df70a895
HADOOP-17111. Replace Guava Optional with Java8+ Optional. Contributed by Ahmed Hussein. 
(cherry picked from commit 639acb6d89)
 2020-07-06 16:09:37 +09:00
Ayush Saxena b8d9218189 HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 12:25:29 +05:30
bilaharith 19fb204011
HADOOP-17086. ABFS: Making the ListStatus response ignore unknown properties. (#2101) 
Contributed by Bilahari T H.

Change-Id: I82e4683fba8481aef2abab7a6a99e5752f6fffa9
 2020-07-03 19:02:21 +01:00
Szilard Nemeth 439c51425e YARN-10330. Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule. Contributed by Peter Bacsko 2020-07-01 17:42:45 +02:00
Szilard Nemeth cfb2084cba YARN-10325. Document max-parallel-apps for Capacity Scheduler. Contributed by Peter Bacsko 2020-07-01 13:40:36 +02:00
Szilard Nemeth d88a6eebf2 YARN-10318. ApplicationHistory Web UI incorrect column indexing. Contributed by Andras Gyori 2020-07-01 13:31:35 +02:00
Akira Ajisaka d572abb818 HADOOP-17090. Increase precommit job timeout from 5 hours to 20 hours. (#2111). Contributed by Akira Ajisaka. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-07-01 15:16:27 +05:30
Abhishek Das 047fb3493a HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 13:01:01 +05:30
Steve Loughran 7de1ac0547
HADOOP-16798. S3A Committer thread pool shutdown problems. (#1963) 
Contributed by Steve Loughran.

Fixes a condition which can cause job commit to fail if a task was
aborted < 60s before the job commit commenced: the task abort
will shut down the thread pool with a hard exit after 60s; the
job commit POST requests would be scheduled through the same pool,
so be interrupted and fail. At present the access is synchronized,
but presumably the executor shutdown code is calling wait() and releasing
locks.

Task abort is triggered from the AM when task attempts succeed but
there are still active speculative task attempts running. Thus it
only surfaces when speculation is enabled and the final tasks are
speculating, which, given they are the stragglers, is not unheard of.

Note: this problem has never been seen in production; it has surfaced
in the hadoop-aws tests on a heavily overloaded desktop

Change-Id: I3b433356d01fcc50d88b4353dbca018484984bc8
 2020-06-30 10:52:56 +01:00
Szilard Nemeth 8b482744e9 YARN-10277. CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy. Contributed by Szilard Nemeth 2020-06-30 11:32:59 +02:00
Akira Ajisaka aa283fc2c2
YARN-10331. Upgrade node.js to 10.21.0. (#2106) 
(cherry picked from commit cd188ea9f0)
 2020-06-30 16:54:59 +09:00
Eric E Payne d7696453a0 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit 74fc13cf91)
 2020-06-29 18:59:52 +00:00
Akira Ajisaka 3e9422d1c7
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac33)
 2020-06-28 16:04:47 +09:00
Virajith Jalaparti ea97fe250c HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a374)
 2020-06-27 16:22:50 -07:00
Uma Maheswara Rao G 81e33d22a0 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155)
 2020-06-27 15:42:36 -07:00
Uma Maheswara Rao G 29a8ee4be6 HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889b)
 2020-06-27 15:42:14 -07:00
Uma Maheswara Rao G 5f67c3f3ca HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa60)
 2020-06-27 15:41:48 -07:00
Uma Maheswara Rao G 3cddd0be29 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a971)
 2020-06-27 15:39:38 -07:00
Ayush Saxena 7b29019eea HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c4)
 2020-06-27 15:39:08 -07:00
Abhishek Das c3bef4906c HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b)
 2020-06-27 15:38:09 -07:00