Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,614 Commits 326 Branches 371 Tags
Commit Graph

19614 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Dhiraj
2c40f20deb
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:07:25 -07:00
Jonathan Hung
aa19cb20ea YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb665761e997bcfc1dc00908f70b069fa2)
(cherry picked from commit 7a323a45aad07eed532d684d6dbe8436ba39c31c)
(cherry picked from commit a80595a6deb3124a3d6d99057e9d5298cd7237d8)
 2020-05-29 10:56:31 -07:00
S O'Donnell
0cd4459f4e HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e2e5cec2ceb28d796c63c83bc8ac506)
 2020-05-29 14:53:59 +01:00
S O'Donnell
1f674e26ba HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:25:35 +01:00
Wilfred Spiegelenburg
0e0d88ca97
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c8562a9e29b00e60d5bde1773c09188)
 2020-05-28 16:38:34 +10:00
Takanobu Asanuma
7168938223 HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f0eb093775f0fe4f39ef4be639ad752)
 2020-05-23 17:35:30 +09:00
Chen Liang
8f53545f9d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:08:08 -07:00
Akira Ajisaka
533e51f97e
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79ed053ce978ac18a2c5706d32e58019f)
 2020-05-18 15:38:01 +09:00
Steve Loughran
31755c91ff Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b1f9dde858ff34df8111e5340115511.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-15 13:49:25 +01:00
Steve Loughran
eebd96636e Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf495f5cf1bb15b1fe6f6e9587d3de0a59.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-15 13:49:13 +01:00
Akira Ajisaka
de4ac41528
Revert "SPNEGO TLS verification" 
This reverts commit 4b7d6667fdc1e252c717fb98a01015f046910672.
 2020-05-13 17:16:15 +09:00
Uma Maheswara Rao G
fda93334dc HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04dfc3d4cefe7a00b63625f60ee232cfe)
 2020-05-12 21:33:49 -07:00
Virajith Jalaparti
bec5d7baa0 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-12 19:37:40 -07:00
Jonathan Hung
b3e9aff5f7 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c1226ec23841b04dd478e8b97f31abbeba)
(cherry picked from commit 564d3211f27c35bf3143a4bd1b3f8eeac2c6b01f)
(cherry picked from commit a7ea55e0156299ec8b80af1f3e681a3a7a31a3b4)
 2020-05-12 10:53:23 -07:00
Ayush Saxena
d24e1bc388 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:45:52 +05:30
Babneet Singh
42997d14a9 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b7be6866eda73223ab6ef1ec78d30cd)
(cherry picked from commit d1e5e393c30ef25212be365651404c1034532338)
(cherry picked from commit 44fdee351dfaecb4012fc47761054cf84d67a667)
 2020-05-11 10:46:21 -07:00
Ayush Saxena
86c15b8287 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:45:13 +05:30
Ayush Saxena
8988f33ac3 HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:41 +05:30
Szilard Nemeth
7e6c5e5ad2 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 
(cherry picked from commit 52e9ee39a12ce91b3a545603dcf1103518ad2920)
 2020-05-07 18:37:02 +00:00
Ahmed Hussein
4e29738f4c YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
 2020-05-06 12:36:30 -05:00
He Xiaoqiao
dbb2809026 HDFS-15272. Backport HDFS-12862 to branch-3.1. Contributed by Xiaoqiao He, Wang XL. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-05-04 18:13:01 -07:00
Inigo Goiri
5612537df0 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 
(cherry picked from commit 4a212242d99071d028b40c2f5d40afb7f337e471)
 2020-05-04 18:03:57 -07:00
Konstantin V Shvachko
8358d7c175 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab991c242b5089a18881aa10abf318ea0)
 2020-05-04 10:50:14 -07:00
Ahmed Hussein
e25ac17e2b YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
 2020-05-04 11:11:14 -05:00
Ayush Saxena
69163f5e5d Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:18:04 +05:30
Eric Yang
4b7d6667fd
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454a5f6ea84f2cf7ac0082c555e2954a7)
 2020-05-02 21:52:06 +09:00
Ayush Saxena
41df5bf835 HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-05-01 14:28:45 +05:30
Gabor Bota
ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Akira Ajisaka
74ec3c001d
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f696aba7fc209874d232baf2a50d547)
 2020-04-28 06:30:38 +09:00
Siyao Meng
b7d9a5279b HADOOP-17014. Upgrade jackson-databind to 2.9.10.4 (#1981) 
(cherry picked from commit 9aef65cdb2a299ba42f1f4a85fcaa4def25462f1)
 2020-04-27 13:36:26 -07:00
Eric E Payne
7e05577c47 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 
(cherry picked from commit e2322e1117a2a3435aabecd49de0ad3c8d2f64b9)
 2020-04-27 20:12:27 +00:00
Mingliang Liu
75354285e6
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:49:29 -07:00
Dhiraj
68dcdad666
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:53:32 -07:00
Wei-Chiu Chuang
6f7a30fd71 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170a062265b7687b31b2a3735827958fb)
(cherry picked from commit d6c48f81009eaf9462cb6ad1ec1cbe609b223cb2)
(cherry picked from commit 9e3b4c9ebd926aa7a5766fcc22239a4951982fc7)
 2020-04-24 11:54:29 -07:00
Mingliang Liu
03ff1d3bd8
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 14:14:19 -07:00
Mingliang Liu
52ea5a0227
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 13:08:03 -07:00
Inigo Goiri
3938aa422e HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena. 
(cherry picked from commit efdfe679d64ce9de4ba6aaf2afa34e180f68d969)
(cherry picked from commit 7d64d44515045bb3dedd91a4a0351bc5a6178a14)
 2020-04-21 09:50:31 -07:00
Akira Ajisaka
e0db6c33d4
HADOOP-16944. Use Yetus 0.12.0 in GitHub PR (#1917) 
(cherry picked from commit 5576915236aba172cb5ab49b43111661590058af)
 2020-04-21 21:13:07 +09:00
Jonathan Hung
6dc1f7e154 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 
(cherry picked from commit 49ae9b2137dddaff1b579586e593a24da21a9e7d)
(cherry picked from commit d1af4e0fae842d46c29562444ebb16905f2caf0d)
 2020-04-17 10:36:26 -07:00
Stephen O'Donnell
0ea8f3a19c HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a9ea9f6e217a0ba158c48da89b3faf2)
(cherry picked from commit aaad947c740c91c36139f3f0569ae78b53bca682)
(cherry picked from commit 041c93a26cd0ff705bea9d6283951702cf2cf1c2)
 2020-04-16 20:19:26 -07:00
Masatake Iwasaki
796bafedba HADOOP-16982. Update Netty to 4.1.48.Final. Contributed by Lisheng Sun. 2020-04-16 11:57:08 +09:00
Eric E Payne
8bcd7cfc0b MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit 11d17417ceba0f1a2944e0c8b4286515c4883889)
 2020-04-13 19:22:56 +00:00
Jonathan Hung
6271a2852e YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 
(cherry picked from commit 57659422abbf6d9bf52e6e27fca775254bb77a56)
(cherry picked from commit e3a52804b03d646f15048c078f8c5292d5cbecfa)
(cherry picked from commit 54599b177c46ed511e096909bed0c4f17bca1fe0)
 2020-04-09 10:44:39 -07:00
Ayush Saxena
2a089e916b HADOOP-16952. Add .diff to gitignore. Contributed by Ayush Saxena. 2020-04-09 19:27:33 +05:30
Akira Ajisaka
d8033bfa96
HADOOP-14836. Upgrade maven-clean-plugin to 3.1.0 (#1933) 
(cherry picked from commit e53d472bb07fc25d6cc5a13af1bd5ffdbf671164)
 2020-04-09 01:50:01 +09:00
Akira Ajisaka
a6371bfdb8
HADOOP-16949. pylint fails in the build environment (#1926) 
(cherry picked from commit e6455cc864d932469610e86644162fa324589067)

Conflicts:
	dev-support/docker/Dockerfile_aarch64
(cherry picked from commit 11aff6c269fed38ba68dd69aaf17adba59a6fec8)
 2020-04-04 16:57:53 +09:00
Ayush Saxena
6a4c3fad66 HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li. 
(cherry picked from commit 62deab17a33cef723d73f8d8b9e37e5bddbc1813)
(cherry picked from commit 6cd9290401735a2c33a0ff0ae7324876ef9615e9)
 2020-03-25 17:14:24 -07:00
Eric Yang
b837431a08 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG

(cherry picked from commit d78854b928bb877f26b11b5b212a100a79941f35)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java

(cherry picked from commit ba6b3a384863b57bc7eeeb736950f544e6ed8d6d)
 2020-03-25 17:14:18 -07:00
Giovanni Matteo Fumarola
4aa7734fc7 HDFS-14006. Refactor name node to allow different token verification implementations. Contributed by CR Hota. 
(cherry picked from commit 00d5e631b596f8712600879366e5283829e7ee5d)
(cherry picked from commit eca7bc7ac4292ff08b9e1ea2e22116d6e58e8b95)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
 2020-03-25 17:13:58 -07:00
Isa Hekmatizadeh
d64f688198 HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 
(cherry picked from commit d9c4f1129c0814ab61fce6ea8baf4b272f84c252)
 2020-03-25 16:52:42 -07:00