Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,622 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

19622 Commits

Author SHA1 Message Date
Kihwal Lee 400c5accd9 HADOOP-16255. Add ChecksumFs.rename(path, path, boolean) 
Contributed by Jungtaek Lim

Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d
(cherry picked from commit bb0b922a71)
 2020-06-08 17:25:15 -05:00
Eric Badger 0799901d48 Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit 7e05577c47.
 2020-06-08 20:12:00 +00:00
Mingliang Liu ac0e928564
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:32:32 -07:00
Mingliang Liu 101ce83f37
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:51:36 -07:00
Toshihiro Suzuki f701d7bdd1 HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit e5a02a7499)
 2020-06-05 12:36:05 +01:00
Akira Ajisaka f30abb3e73
HADOOP-17056. Addendum patch: Fix typo 
(cherry picked from commit 5157118bd7)
(cherry picked from commit 59e7275beb)
 2020-06-04 16:43:23 +09:00
Akira Ajisaka de4f1aabfa
HADOOP-17056. shelldoc fails in hadoop-common. (#2045) 
In the docker build image, skip GPG verification when downloading
Yetus tarball via yetus-wrapper.

(cherry picked from commit 9c290c08db)

 Conflicts:
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64

(cherry picked from commit aa19a2af17)
 2020-06-04 16:41:22 +09:00
Akira Ajisaka 30c639a025
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 
(cherry picked from commit 704409d53b)
 2020-06-04 06:07:54 +09:00
Dhiraj 2c40f20deb
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:07:25 -07:00
Jonathan Hung aa19cb20ea YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb66)
(cherry picked from commit 7a323a45aa)
(cherry picked from commit a80595a6deb3124a3d6d99057e9d5298cd7237d8)
 2020-05-29 10:56:31 -07:00
S O'Donnell 0cd4459f4e HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e)
 2020-05-29 14:53:59 +01:00
S O'Donnell 1f674e26ba HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:25:35 +01:00
Wilfred Spiegelenburg 0e0d88ca97
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c)
 2020-05-28 16:38:34 +10:00
Takanobu Asanuma 7168938223 HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f)
 2020-05-23 17:35:30 +09:00
Chen Liang 8f53545f9d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:08:08 -07:00
Akira Ajisaka 533e51f97e
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79e)
 2020-05-18 15:38:01 +09:00
Steve Loughran 31755c91ff Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-15 13:49:25 +01:00
Steve Loughran eebd96636e Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf49.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-15 13:49:13 +01:00
Akira Ajisaka de4ac41528
Revert "SPNEGO TLS verification" 
This reverts commit 4b7d6667fd.
 2020-05-13 17:16:15 +09:00
Uma Maheswara Rao G fda93334dc HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-12 21:33:49 -07:00
Virajith Jalaparti bec5d7baa0 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-12 19:37:40 -07:00
Jonathan Hung b3e9aff5f7 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c122)
(cherry picked from commit 564d3211f2)
(cherry picked from commit a7ea55e015)
 2020-05-12 10:53:23 -07:00
Ayush Saxena d24e1bc388 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:45:52 +05:30
Babneet Singh 42997d14a9 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b)
(cherry picked from commit d1e5e393c3)
(cherry picked from commit 44fdee351d)
 2020-05-11 10:46:21 -07:00
Ayush Saxena 86c15b8287 HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:45:13 +05:30
Ayush Saxena 8988f33ac3 HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:41 +05:30
Szilard Nemeth 7e6c5e5ad2 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 
(cherry picked from commit 52e9ee39a1)
 2020-05-07 18:37:02 +00:00
Ahmed Hussein 4e29738f4c YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb401)
 2020-05-06 12:36:30 -05:00
He Xiaoqiao dbb2809026 HDFS-15272. Backport HDFS-12862 to branch-3.1. Contributed by Xiaoqiao He, Wang XL. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-05-04 18:13:01 -07:00
Inigo Goiri 5612537df0 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 
(cherry picked from commit 4a212242d9)
 2020-05-04 18:03:57 -07:00
Konstantin V Shvachko 8358d7c175 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab9)
 2020-05-04 10:50:14 -07:00
Ahmed Hussein e25ac17e2b YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d)
 2020-05-04 11:11:14 -05:00
Ayush Saxena 69163f5e5d Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:18:04 +05:30
Eric Yang 4b7d6667fd
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:52:06 +09:00
Ayush Saxena 41df5bf835 HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-05-01 14:28:45 +05:30
Gabor Bota ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Akira Ajisaka 74ec3c001d
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f)
 2020-04-28 06:30:38 +09:00
Siyao Meng b7d9a5279b HADOOP-17014. Upgrade jackson-databind to 2.9.10.4 (#1981) 
(cherry picked from commit 9aef65cdb2)
 2020-04-27 13:36:26 -07:00
Eric E Payne 7e05577c47 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 
(cherry picked from commit e2322e1117)
 2020-04-27 20:12:27 +00:00
Mingliang Liu 75354285e6
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:49:29 -07:00
Dhiraj 68dcdad666
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:53:32 -07:00
Wei-Chiu Chuang 6f7a30fd71 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170)
(cherry picked from commit d6c48f8100)
(cherry picked from commit 9e3b4c9ebd)
 2020-04-24 11:54:29 -07:00
Mingliang Liu 03ff1d3bd8
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 14:14:19 -07:00
Mingliang Liu 52ea5a0227
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 13:08:03 -07:00
Inigo Goiri 3938aa422e HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena. 
(cherry picked from commit efdfe679d6)
(cherry picked from commit 7d64d44515)
 2020-04-21 09:50:31 -07:00
Akira Ajisaka e0db6c33d4
HADOOP-16944. Use Yetus 0.12.0 in GitHub PR (#1917) 
(cherry picked from commit 5576915236)
 2020-04-21 21:13:07 +09:00
Jonathan Hung 6dc1f7e154 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 
(cherry picked from commit 49ae9b2137)
(cherry picked from commit d1af4e0fae)
 2020-04-17 10:36:26 -07:00
Stephen O'Donnell 0ea8f3a19c HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a)
(cherry picked from commit aaad947c74)
(cherry picked from commit 041c93a26c)
 2020-04-16 20:19:26 -07:00
Masatake Iwasaki 796bafedba HADOOP-16982. Update Netty to 4.1.48.Final. Contributed by Lisheng Sun. 2020-04-16 11:57:08 +09:00
Eric E Payne 8bcd7cfc0b MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit 11d17417ce)
 2020-04-13 19:22:56 +00:00