Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,662 Commits 326 Branches 371 Tags
Commit Graph

21662 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Szilard Nemeth
54c89ffad4 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-05 09:49:54 +02:00
Akira Ajisaka
59e7275beb
HADOOP-17056. Addendum patch: Fix typo 
(cherry picked from commit 5157118bd7f3448949da885e323c163828c35aee)
 2020-06-04 16:42:59 +09:00
Akira Ajisaka
aa19a2af17
HADOOP-17056. shelldoc fails in hadoop-common. (#2045) 
In the docker build image, skip GPG verification when downloading
Yetus tarball via yetus-wrapper.

(cherry picked from commit 9c290c08db4361de29f392b0569312c2623b8321)

 Conflicts:
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2020-06-04 16:40:34 +09:00
Akira Ajisaka
d9529596f4
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 
(cherry picked from commit 704409d53bf7ebf717a3c2e988ede80f623bbad3)
 2020-06-04 06:06:28 +09:00
Dhiraj
462a68a69f
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:06:32 -07:00
Jonathan Hung
f31146bc1f YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb665761e997bcfc1dc00908f70b069fa2)
(cherry picked from commit 7a323a45aad07eed532d684d6dbe8436ba39c31c)
 2020-05-29 10:43:33 -07:00
S O'Donnell
d245a4d7fe HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e2e5cec2ceb28d796c63c83bc8ac506)
 2020-05-29 14:51:45 +01:00
S O'Donnell
5b6c2b366e HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:10:50 +01:00
Wilfred Spiegelenburg
e2713d5788
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c8562a9e29b00e60d5bde1773c09188)
 2020-05-28 13:39:57 +10:00
Takanobu Asanuma
d6de530bbb HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f0eb093775f0fe4f39ef4be639ad752)
 2020-05-23 17:31:59 +09:00
Masatake Iwasaki
77e69a73da HADOOP-17040. Fix intermittent failure of ITestBlockingThreadPoolExecutorService. (#2020) 
(cherry picked from commit 968531463375ebf29ba3186c13b5f8685df10d25)
 2020-05-22 21:30:57 +09:00
Chen Liang
1813d25bf2 HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:05:47 -07:00
Akira Ajisaka
777e36448c
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79ed053ce978ac18a2c5706d32e58019f)
 2020-05-18 15:37:42 +09:00
Steve Loughran
168c597ece Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf495f5cf1bb15b1fe6f6e9587d3de0a59.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-15 13:47:22 +01:00
Steve Loughran
7a8b8c3588 Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b1f9dde858ff34df8111e5340115511.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-15 13:46:07 +01:00
Akira Ajisaka
dfb094d581
Revert "SPNEGO TLS verification" 
This reverts commit 9db6e96db3cd4a11075afa0c0b58476cd0e26b8a.
 2020-05-13 17:15:46 +09:00
Jonathan Hung
a7ea55e015 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c1226ec23841b04dd478e8b97f31abbeba)
(cherry picked from commit 564d3211f27c35bf3143a4bd1b3f8eeac2c6b01f)
 2020-05-12 10:52:58 -07:00
Ayush Saxena
b706ac11f7 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:44:56 +05:30
Uma Maheswara Rao G
0d0aac1a2a HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04dfc3d4cefe7a00b63625f60ee232cfe)
 2020-05-11 15:29:26 -07:00
Virajith Jalaparti
e5e824bd6f HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-11 12:50:14 -07:00
Babneet Singh
44fdee351d HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b7be6866eda73223ab6ef1ec78d30cd)
(cherry picked from commit d1e5e393c30ef25212be365651404c1034532338)
 2020-05-11 10:46:10 -07:00
Ayush Saxena
06f43fc628 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:43 +05:30
Ayush Saxena
c3fffa8451 HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:24 +05:30
Szilard Nemeth
d345994468 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 
(cherry picked from commit 52e9ee39a12ce91b3a545603dcf1103518ad2920)
 2020-05-07 18:17:06 +00:00
Masatake Iwasaki
89696b66e7 HADOOP-17025. Fix invalid metastore configuration in S3GuardTool tests. (#1994) 
(cherry picked from commit 99840aaba662e7bb6187206b74e35132102a3b38)

 Conflicts:
	hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
 2020-05-07 12:11:37 +09:00
Ahmed Hussein
7740b88ee9 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
 2020-05-06 12:32:36 -05:00
Inigo Goiri
53f6d26500 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 
(cherry picked from commit 4a212242d99071d028b40c2f5d40afb7f337e471)
 2020-05-04 18:04:07 -07:00
Konstantin V Shvachko
e95a4ddbb1 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab991c242b5089a18881aa10abf318ea0)
 2020-05-04 10:47:07 -07:00
Ahmed Hussein
b23a585cb1 YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
 2020-05-04 10:49:45 -05:00
Ayush Saxena
d451c9e2a4 Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:44 +05:30
Eric Yang
9db6e96db3
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454a5f6ea84f2cf7ac0082c555e2954a7)
 2020-05-02 21:51:41 +09:00
Ayush Saxena
50ba1949ee HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-05-01 14:27:24 +05:30
Szilard Nemeth
1dabbd5006 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 2020-04-28 17:52:14 +02:00
Akira Ajisaka
5154f6eef4
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f696aba7fc209874d232baf2a50d547)
 2020-04-28 06:19:17 +09:00
Siyao Meng
9aef65cdb2
HADOOP-17014. Upgrade jackson-databind to 2.9.10.4 (#1981) 2020-04-27 13:34:33 -07:00
Eric E Payne
6bf255b4ea MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 
(cherry picked from commit e2322e1117a2a3435aabecd49de0ad3c8d2f64b9)
 2020-04-27 19:56:48 +00:00
Szilard Nemeth
f445487d50 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-27 09:59:13 +02:00
Mingliang Liu
57c1e7ad9b
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:44:27 -07:00
Dhiraj
6225fce50d
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:08:32 -07:00
Wei-Chiu Chuang
9e3b4c9ebd HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170a062265b7687b31b2a3735827958fb)
(cherry picked from commit d6c48f81009eaf9462cb6ad1ec1cbe609b223cb2)
 2020-04-24 11:54:20 -07:00
Steve Loughran
204d54005a HADOOP-16117. Update AWS SDK to 1.11.563. 
Contributed by Steve Loughran.

Change-Id: I7c46ed2a6378e1370f567acf4cdcfeb93e43fa13
 2020-04-24 10:46:24 +01:00
Szilard Nemeth
0dbb02a76c YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:31:33 +02:00
Szilard Nemeth
3b67dc24aa YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-04-24 11:15:53 +02:00
Masatake Iwasaki
5ebfb7203a HADOOP-16647. Support OpenSSL 1.1.1 LTS. Contributed by Rakesh Radhakrishnan. 
(cherry picked from commit 8f8be6b92a32982c2fd5be4647ebe1610a533dc3)
 2020-04-24 15:38:11 +09:00
Masatake Iwasaki
83c4f8b9a0 HADOOP-16739. Fix native build failure of hadoop-pipes on CentOS 8. 2020-04-24 15:38:11 +09:00
Akira Ajisaka
7b036c512f
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 
(cherry picked from commit 9827ff2961e249e8be9023f39d91e1c0fbd23d61)
 2020-04-24 10:47:36 +09:00
Wei-Chiu Chuang
48f1c8ffb6 Revert "YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja" 
This reverts commit a2067aafa941344c2c1713ba835a107f7231c2cc.
 2020-04-23 12:37:21 -07:00
Szilard Nemeth
c81844d8a5 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:57:18 +02:00
Szilard Nemeth
764fa92c9f YARN-9997. Code cleanup in ZKConfigurationStore. Contributed by Andras Gyori 2020-04-23 14:51:07 +02:00
Mingliang Liu
e154d9d9f7
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:56:15 -07:00