Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,499 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21499 Commits

Author SHA1 Message Date
Siyao Meng 0bd0cb1b06 HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9)
 2020-07-09 16:31:43 -07:00
Uma Maheswara Rao G 7607f24c9e HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5)
 2020-07-09 16:29:02 -07:00
Masatake Iwasaki 936dece92b YARN-10347. Fix double locking in CapacityScheduler#reinitialize in branch-3.1. 
(cherry picked from commit 4fa8055aa4)
 2020-07-09 14:19:22 +09:00
Masatake Iwasaki aeb3c4a362 HADOOP-17120. Fix failure of docker image creation due to pip2 install error. (#2130) 
(cherry picked from commit 53cfe60904)
 2020-07-09 02:48:33 +00:00
Shanyu Zhao 8d4028864d HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-07 11:53:29 +08:00
Madhusoodan Pataki dfb74a3eeb HADOOP-17081. MetricsSystem doesn't start the sink adapters on restart (#2089) 
Contributed by Madhusoodan P
 2020-07-06 16:29:09 +01:00
Ayush Saxena 642aafb106 HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 12:25:54 +05:30
Akira Ajisaka fef5b3c1c7 HADOOP-17090. Increase precommit job timeout from 5 hours to 20 hours. (#2111). Contributed by Akira Ajisaka. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-07-01 15:16:45 +05:30
Abhishek Das b7d54c3dbf HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 13:01:31 +05:30
Eric E Payne e6794f2fc4 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 2020-06-29 19:21:04 +00:00
Akira Ajisaka 37e0a1208c
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac33)
 2020-06-28 16:16:49 +09:00
Virajith Jalaparti 704b273ec8 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a374)
 2020-06-27 16:25:04 -07:00
Uma Maheswara Rao G d030f4b2a6 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155)
 2020-06-27 15:56:10 -07:00
Uma Maheswara Rao G 80111fe5bb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889b)
 2020-06-27 15:55:41 -07:00
Uma Maheswara Rao G 6c22210baa HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa60)
 2020-06-27 15:55:11 -07:00
Uma Maheswara Rao G 4a9ec4d143 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a971)
 2020-06-27 15:54:37 -07:00
Ayush Saxena fc9445abbd HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c4)
 2020-06-27 15:54:13 -07:00
Abhishek Das db5c32d2a5 HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b)
 2020-06-27 15:53:38 -07:00
Uma Maheswara Rao G aa8de2f43b HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def95)
(cherry picked from commit 120ee793fc)
 2020-06-27 15:53:07 -07:00
Ayush Saxena e80421820a HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 
(cherry picked from commit cc671b16f7)
(cherry picked from commit bee2846bee)
 2020-06-27 15:52:47 -07:00
Uma Maheswara Rao G e7cead1143 HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 76fa0222f0)
(cherry picked from commit 418580446b)
 2020-06-27 15:52:18 -07:00
Uma Maheswara Rao G 852ee71354 HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd)
(cherry picked from commit 0b5e202614)
 2020-06-27 15:51:51 -07:00
Uma Maheswara Rao G 84ceb6d520 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b)
(cherry picked from commit 8e71e85af7)
 2020-06-27 15:51:28 -07:00
Abhishek Das 7cf9601987 HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec74453)
(cherry picked from commit 5b248de42d)
 2020-06-27 15:50:41 -07:00
Uma Maheswara Rao G 3dcc9aed4d HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d9)
(cherry picked from commit 544996c857)
 2020-06-27 15:49:48 -07:00
S O'Donnell 240974932b HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 
(cherry picked from commit 123777823e)
 2020-06-18 12:44:42 +01:00
Eric Yang e9cf454ac2
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0)
 2020-06-15 11:12:40 +09:00
Takanobu Asanuma 695d20615d HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 
(cherry picked from commit f41a144077)
 2020-06-15 09:18:22 +09:00
Eric Badger 7363931942 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan
 2020-06-12 19:08:36 +00:00
Szilard Nemeth 30d7a06686 YARN-10295. CapacityScheduler NPE can cause apps to get stuck without resources. Contributed by Benjamin Teke 2020-06-10 18:16:21 +02:00
Szilard Nemeth 8bef26a607 YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-10 18:01:20 +02:00
Eric E Payne 034d458511 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db302)
 2020-06-09 21:09:11 +00:00
Ayush Saxena 452d801f1b HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:48:48 +05:30
Eric Badger d07e00008c Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit 6bf255b4ea.
 2020-06-08 20:19:14 +00:00
Mingliang Liu 61106467e3
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:32:07 -07:00
Mingliang Liu 0eee1c88ec
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:39:09 -07:00
Toshihiro Suzuki e5a02a7499 HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2020-06-05 12:26:52 +01:00
Szilard Nemeth 54c89ffad4 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-05 09:49:54 +02:00
Akira Ajisaka 59e7275beb
HADOOP-17056. Addendum patch: Fix typo 
(cherry picked from commit 5157118bd7)
 2020-06-04 16:42:59 +09:00
Akira Ajisaka aa19a2af17
HADOOP-17056. shelldoc fails in hadoop-common. (#2045) 
In the docker build image, skip GPG verification when downloading
Yetus tarball via yetus-wrapper.

(cherry picked from commit 9c290c08db)

 Conflicts:
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2020-06-04 16:40:34 +09:00
Akira Ajisaka d9529596f4
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 
(cherry picked from commit 704409d53b)
 2020-06-04 06:06:28 +09:00
Dhiraj 462a68a69f
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:06:32 -07:00
Jonathan Hung f31146bc1f YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb66)
(cherry picked from commit 7a323a45aa)
 2020-05-29 10:43:33 -07:00
S O'Donnell d245a4d7fe HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e)
 2020-05-29 14:51:45 +01:00
S O'Donnell 5b6c2b366e HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:10:50 +01:00
Wilfred Spiegelenburg e2713d5788
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c)
 2020-05-28 13:39:57 +10:00
Takanobu Asanuma d6de530bbb HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f)
 2020-05-23 17:31:59 +09:00
Masatake Iwasaki 77e69a73da HADOOP-17040. Fix intermittent failure of ITestBlockingThreadPoolExecutorService. (#2020) 
(cherry picked from commit 9685314633)
 2020-05-22 21:30:57 +09:00
Chen Liang 1813d25bf2 HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:05:47 -07:00
Akira Ajisaka 777e36448c
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79e)
 2020-05-18 15:37:42 +09:00