Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,289 Commits 326 Branches 371 Tags
Commit Graph

21289 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akira Ajisaka
37e0a1208c
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac3370e220995bad0ae8b59d962c8d30a7)
 2020-06-28 16:16:49 +09:00
Virajith Jalaparti
704b273ec8 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a37404e9defda13a5bffe5609e72466e46)
 2020-06-27 16:25:04 -07:00
Uma Maheswara Rao G
d030f4b2a6 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155496cf9d82e121dd1b65d0072312197)
 2020-06-27 15:56:10 -07:00
Uma Maheswara Rao G
80111fe5bb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889bbeabc73c95a4c83f0cd204365ff410)
 2020-06-27 15:55:41 -07:00
Uma Maheswara Rao G
6c22210baa HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa6015253866ccc0ccc7247ad7024c0730)
 2020-06-27 15:55:11 -07:00
Uma Maheswara Rao G
4a9ec4d143 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
 2020-06-27 15:54:37 -07:00
Ayush Saxena
fc9445abbd HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c498531bf3c011f1b0ace8eddddf61f2)
 2020-06-27 15:54:13 -07:00
Abhishek Das
db5c32d2a5 HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b658b2a1f216fbedc65938d9b6ca6e9)
 2020-06-27 15:53:38 -07:00
Uma Maheswara Rao G
aa8de2f43b HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def959fab6b8b7ffff66410bcd240feee13)
(cherry picked from commit 120ee793fc4bcbf9d1945d5e38e3ad5b2b290a0e)
 2020-06-27 15:53:07 -07:00
Ayush Saxena
e80421820a HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 
(cherry picked from commit cc671b16f7b0b7c1ed7b41b96171653dc43cf670)
(cherry picked from commit bee2846bee4ae676bdc14585f8a3927a9dd7df37)
 2020-06-27 15:52:47 -07:00
Uma Maheswara Rao G
e7cead1143 HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 76fa0222f0d2e2d92b4a1eedba8b3e38002e8c23)
(cherry picked from commit 418580446b65be3a0674762e76fc2cb9a1e5629a)
 2020-06-27 15:52:18 -07:00
Uma Maheswara Rao G
852ee71354 HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd0b4e92f3f89f79543acc23792bb69c)
(cherry picked from commit 0b5e202614f0bc20a0db6656f924fa4d2741d00c)
 2020-06-27 15:51:51 -07:00
Uma Maheswara Rao G
84ceb6d520 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b64b7c6432da4cc32881aba85f94ea1)
(cherry picked from commit 8e71e85af70c17f2350f794f8bc2475eb1e3acea)
 2020-06-27 15:51:28 -07:00
Abhishek Das
7cf9601987 HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec7445345eb94c6741d416814a4eac319f0a6)
(cherry picked from commit 5b248de42d2ae42710531a1514a21d60a1fca4b2)
 2020-06-27 15:50:41 -07:00
Uma Maheswara Rao G
3dcc9aed4d HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d98827c7926a34cda27aa7bcfd3f36c1)
(cherry picked from commit 544996c85702af7ae241ef2f18e2597e2b4050be)
 2020-06-27 15:49:48 -07:00
S O'Donnell
240974932b HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 
(cherry picked from commit 123777823edc98553fcef61f1913ab6e4cd5aa9a)
 2020-06-18 12:44:42 +01:00
Eric Yang
e9cf454ac2
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0406380e469c76ed2e41022a6372dd7)
 2020-06-15 11:12:40 +09:00
Takanobu Asanuma
695d20615d HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 
(cherry picked from commit f41a144077fc0e2d32072e0d088c1abd1897cee5)
 2020-06-15 09:18:22 +09:00
Eric Badger
7363931942 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan
 2020-06-12 19:08:36 +00:00
Szilard Nemeth
30d7a06686 YARN-10295. CapacityScheduler NPE can cause apps to get stuck without resources. Contributed by Benjamin Teke 2020-06-10 18:16:21 +02:00
Szilard Nemeth
8bef26a607 YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-10 18:01:20 +02:00
Eric E Payne
034d458511 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db3022705635580c4d2f8b0abde109f954f)
 2020-06-09 21:09:11 +00:00
Ayush Saxena
452d801f1b HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:48:48 +05:30
Eric Badger
d07e00008c Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit 6bf255b4ead3130d45a8483245b65e742a19642f.
 2020-06-08 20:19:14 +00:00
Mingliang Liu
61106467e3
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:32:07 -07:00
Mingliang Liu
0eee1c88ec
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:39:09 -07:00
Toshihiro Suzuki
e5a02a7499 HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c5256c44911ba57b4898e01d786d836)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2020-06-05 12:26:52 +01:00
Szilard Nemeth
54c89ffad4 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-05 09:49:54 +02:00
Akira Ajisaka
59e7275beb
HADOOP-17056. Addendum patch: Fix typo 
(cherry picked from commit 5157118bd7f3448949da885e323c163828c35aee)
 2020-06-04 16:42:59 +09:00
Akira Ajisaka
aa19a2af17
HADOOP-17056. shelldoc fails in hadoop-common. (#2045) 
In the docker build image, skip GPG verification when downloading
Yetus tarball via yetus-wrapper.

(cherry picked from commit 9c290c08db4361de29f392b0569312c2623b8321)

 Conflicts:
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2020-06-04 16:40:34 +09:00
Akira Ajisaka
d9529596f4
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 
(cherry picked from commit 704409d53bf7ebf717a3c2e988ede80f623bbad3)
 2020-06-04 06:06:28 +09:00
Dhiraj
462a68a69f
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:06:32 -07:00
Jonathan Hung
f31146bc1f YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb665761e997bcfc1dc00908f70b069fa2)
(cherry picked from commit 7a323a45aad07eed532d684d6dbe8436ba39c31c)
 2020-05-29 10:43:33 -07:00
S O'Donnell
d245a4d7fe HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e2e5cec2ceb28d796c63c83bc8ac506)
 2020-05-29 14:51:45 +01:00
S O'Donnell
5b6c2b366e HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:10:50 +01:00
Wilfred Spiegelenburg
e2713d5788
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c8562a9e29b00e60d5bde1773c09188)
 2020-05-28 13:39:57 +10:00
Takanobu Asanuma
d6de530bbb HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f0eb093775f0fe4f39ef4be639ad752)
 2020-05-23 17:31:59 +09:00
Masatake Iwasaki
77e69a73da HADOOP-17040. Fix intermittent failure of ITestBlockingThreadPoolExecutorService. (#2020) 
(cherry picked from commit 968531463375ebf29ba3186c13b5f8685df10d25)
 2020-05-22 21:30:57 +09:00
Chen Liang
1813d25bf2 HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:05:47 -07:00
Akira Ajisaka
777e36448c
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79ed053ce978ac18a2c5706d32e58019f)
 2020-05-18 15:37:42 +09:00
Steve Loughran
168c597ece Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf495f5cf1bb15b1fe6f6e9587d3de0a59.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-15 13:47:22 +01:00
Steve Loughran
7a8b8c3588 Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b1f9dde858ff34df8111e5340115511.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-15 13:46:07 +01:00
Akira Ajisaka
dfb094d581
Revert "SPNEGO TLS verification" 
This reverts commit 9db6e96db3cd4a11075afa0c0b58476cd0e26b8a.
 2020-05-13 17:15:46 +09:00
Jonathan Hung
a7ea55e015 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c1226ec23841b04dd478e8b97f31abbeba)
(cherry picked from commit 564d3211f27c35bf3143a4bd1b3f8eeac2c6b01f)
 2020-05-12 10:52:58 -07:00
Ayush Saxena
b706ac11f7 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:44:56 +05:30
Uma Maheswara Rao G
0d0aac1a2a HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04dfc3d4cefe7a00b63625f60ee232cfe)
 2020-05-11 15:29:26 -07:00
Virajith Jalaparti
e5e824bd6f HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-11 12:50:14 -07:00
Babneet Singh
44fdee351d HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b7be6866eda73223ab6ef1ec78d30cd)
(cherry picked from commit d1e5e393c30ef25212be365651404c1034532338)
 2020-05-11 10:46:10 -07:00
Ayush Saxena
06f43fc628 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:43 +05:30
Ayush Saxena
c3fffa8451 HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:24 +05:30