Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,720 Commits 326 Branches 371 Tags
Commit Graph

19720 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akira Ajisaka
d8033bfa96
HADOOP-14836. Upgrade maven-clean-plugin to 3.1.0 (#1933) 
(cherry picked from commit e53d472bb07fc25d6cc5a13af1bd5ffdbf671164)
 2020-04-09 01:50:01 +09:00
Akira Ajisaka
a6371bfdb8
HADOOP-16949. pylint fails in the build environment (#1926) 
(cherry picked from commit e6455cc864d932469610e86644162fa324589067)

Conflicts:
	dev-support/docker/Dockerfile_aarch64
(cherry picked from commit 11aff6c269fed38ba68dd69aaf17adba59a6fec8)
 2020-04-04 16:57:53 +09:00
Ayush Saxena
6a4c3fad66 HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li. 
(cherry picked from commit 62deab17a33cef723d73f8d8b9e37e5bddbc1813)
(cherry picked from commit 6cd9290401735a2c33a0ff0ae7324876ef9615e9)
 2020-03-25 17:14:24 -07:00
Eric Yang
b837431a08 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG

(cherry picked from commit d78854b928bb877f26b11b5b212a100a79941f35)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java

(cherry picked from commit ba6b3a384863b57bc7eeeb736950f544e6ed8d6d)
 2020-03-25 17:14:18 -07:00
Giovanni Matteo Fumarola
4aa7734fc7 HDFS-14006. Refactor name node to allow different token verification implementations. Contributed by CR Hota. 
(cherry picked from commit 00d5e631b596f8712600879366e5283829e7ee5d)
(cherry picked from commit eca7bc7ac4292ff08b9e1ea2e22116d6e58e8b95)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
 2020-03-25 17:13:58 -07:00
Isa Hekmatizadeh
d64f688198 HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 
(cherry picked from commit d9c4f1129c0814ab61fce6ea8baf4b272f84c252)
 2020-03-25 16:52:42 -07:00
Ayush Saxena
4891f24a94 HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun. 
(cherry picked from commit 6191d4b4a0919863fda78e549ab6c60022e3ebc2)
(cherry picked from commit 1e3b0df6abcc1252907c41aaedb3e7e257bce497)
 2020-03-25 16:42:09 -07:00
Yiqun Lin
f1a19b7a3f HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3)
 2020-03-25 16:31:54 -07:00
Jonathan Hung
9c6dd8c83a YARN-10200. Add number of containers to RMAppManager summary 
(cherry picked from commit 2de0572cdc1c6fdbfaab108b169b2d5b0c077e86)
(cherry picked from commit 5d3fb0ebe9d3f3395320b82a76194ba6fad01e00)
 2020-03-25 10:33:18 -07:00
Sean Mackrory
dde8417972 HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota. 2020-03-23 13:53:18 -07:00
Sean Mackrory
770e2aaf83 HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota. 
(cherry picked from commit b089a06793d94d42b7da1b7566e366ceb748e081)
 2020-03-23 13:52:08 -07:00
Wei-Chiu Chuang
61915fbd83 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
(cherry picked from commit 096533c2dc0afd51367030725d797480a22ba7e2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties

(cherry picked from commit 1d55e1e229d835a6743523b9cf4c1e97b4259128)
 2020-03-18 09:59:12 -07:00
Surendra Singh Lilhore
a3224ff799 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:59:00 +05:30
Ayush Saxena
8566c65917 HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:31:58 +05:30
Steve Loughran
caec6a1945 HADOOP-16775. DistCp reuses the same temp file within the task for different files. 
Contributed by Amir Shenavandeh.

This avoids overwrite consistency issues with S3 and other stores

Change-Id: Ic4d05ef3397e963ba28fd9f775bb362b0da36ad9
 2020-03-13 19:34:50 +00:00
S O'Donnell
4e1021975b HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:47:57 +00:00
Eric E Payne
80e394e81b YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit ede05b19d1723147430fc426161326d46698507f)
 2020-03-10 15:25:10 +00:00
Brahma Reddy Battula
e829631eed HADOOP-16871. Upgrade Netty version to 4.1.45.Final to handle CVE-2019-20444,CVE-2019-16869 
(cherry picked from commit c6b8a3038646697b77f6db54a2ef6266a9fc7888)
(cherry picked from commit 74fa55afc3d333d7d99397754f45b3fc54bc0f4a)

 Conflicts:
	hadoop-project/pom.xml
 2020-03-09 18:22:18 -07:00
Wei-Chiu Chuang
7bb5bf0e4e HADOOP-16643. Update netty4 to the latest 4.1.42. Contributed by Lisheng Sun. 
(cherry picked from commit 85af77c75768416db24ca506fd1704ce664ca92f)
 2020-03-09 17:58:15 -07:00
Weiwei Yang
79e5633867 HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 
(cherry picked from commit 6dfe00c71eb3721e9be3fc42349a81c4b013ada1)
 2020-03-08 22:01:59 -07:00
John Zhuge
c3249fa1ea
HADOOP-14206. TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature. Contributed by Jim Brennan. 
(cherry picked from commit 999096d82e6cc24023a77a69509af6820913f942)
 2020-03-08 00:02:02 -08:00
Siyao Meng
57783e7e24 HADOOP-16891. Upgrade jackson-databind to 2.9.10.3 (#1865) 
(cherry picked from commit e36b27260845c2eeb2211d01235cc6d3578b1942)
(cherry picked from commit 8a55178f4983f95e207656a6f81142f510c0cd7a)
 2020-02-28 17:03:38 -08:00
Wei-Chiu Chuang
58b025c8f4 HADOOP-16882. Update jackson-databind to 2.9.10.2 in branch-3.1, branch-2.10. Contributed by Lisheng Sun. 2020-02-28 16:24:38 -08:00
Surendra Singh Lilhore
64eb5b2709 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:47:54 +05:30
Mukund Thakur
ce6614565e HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running 
This uses the length of the file known at the start of the copy to determine the amount of data to copy.

* If a file is appended to during the copy, the original bytes are copied.
* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,
  distcp will now fail. Until now these failures were not detected.

Contributed by Mukund Thakur.

Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884
(cherry picked from commit 51c64b357d4bd1a0038e61df3d4b8ea0a3ad7449)
 2020-02-27 16:37:36 -08:00
Ayush Saxena
354e63b366 HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 
(cherry picked from commit df622cf4a32ee172ded6c4b3b97a1e49befc4f10)
 2020-02-27 16:37:30 -08:00
Masatake Iwasaki
abee7402e4 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 037ec8cfb1406ea3a8225a1b6306c2e78440353b)
 2020-02-27 16:37:23 -08:00
Konstantin V Shvachko
4af188587f HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ecbcb058b8bc0fbc3903acb56814c6d9608bc396)
 2020-02-27 16:02:30 -08:00
Ayush Saxena
7bfa123d58 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 
(cherry picked from commit b182dd809cf42aed66806901631ad6062ca0820c)
 2020-02-27 14:55:50 -08:00
Inigo Goiri
4924622e6e YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 
(cherry picked from commit a43510e21d01e6c78e98e7ad9469cbea70a66466)
 2020-02-27 13:20:17 -08:00
Istvan Fajth
b5022b0515 HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 
(cherry picked from commit 57aa048516f5c5fe02441d213b52ce1bbeddf823)
(cherry picked from commit e42ac486e7eecb6a24ac95f1ceaf61d24060adef)
 2020-02-27 07:49:45 -08:00
Kihwal Lee
b92477c638 HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed 
by Ahmed Hussein.

(cherry picked from commit 27cfda708ef66dfbe5f52a5f1e716298a294f3f7)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
 2020-02-27 09:45:12 -06:00
Elixir Kook
9ccefe2262
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 
(cherry picked from commit d608e94f9208354897a59dc8e533535b1b8a30b9)
 2020-02-26 17:32:47 +09:00
Stephen O'Donnell
8aaa8d1b71 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b9907b590ed2d73a0d68f079c7f4d754b1)
(cherry picked from commit 2377649cdb1307a72a946b95a9a04f316c6483ef)
 2020-02-25 13:29:15 -08:00
Takanobu Asanuma
241c5a14c5 HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li. 
(cherry picked from commit 5cbc4c54611f062690c3bfcf78d09d37c78ffd12)
 2020-02-25 11:10:22 +09:00
Wei-Chiu Chuang
268e35d3ae HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
(cherry picked from commit f9437b1817e09c5a3ee7b378a0f4faee9b853315)
 2020-02-24 13:37:59 -08:00
Ayush Saxena
d318b48e12 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:52 +05:30
Konstantin V Shvachko
cd0525054d HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 6f84269bcd5cdb08ca68b2d8276f66d34a2a7a0d)
 2020-02-21 10:56:31 -08:00
Masatake Iwasaki
8095e3b149 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java

(cherry picked from commit 610805ec7245769aebb36e52725522c42cb3dd88)
 2020-02-21 13:51:26 +09:00
Akira Ajisaka
3f8ac06cef
HADOOP-16869. Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure (#1855) 
(cherry picked from commit 7f35676f90f0730b8c9844cf00ee5a943f80d48d)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit b52f2012858d99c8959e63034dec947454363674)
 2020-02-21 12:24:41 +09:00
Wei-Chiu Chuang
b28347d282 HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze. 
(cherry picked from commit cb3f3cca01d6ab9f4befb1bcd31f384c37c0231a)
(cherry picked from commit d411a849e10910d24cf1bbc8d64405470e00f510)
 2020-02-18 21:54:00 -08:00
Sunil G
a6124cd2b8 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 
(cherry picked from commit 6526f95bd281fc011f8776d21ff933087c5924de)
 2020-02-19 11:18:53 +05:30
Kihwal Lee
c54051def5 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71cba068e7abf1c76ad629bfeec10852)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-17 16:07:03 -06:00
xiaoxiaopan118
b59fa3e571
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 
(cherry picked from commit 439d935e1df601ed998521443fbe6752040e7a84)
 2020-02-17 15:58:35 +09:00
Ayush Saxena
2190fb1fcf HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-16 10:09:40 +05:30
Ayush Saxena
cf5a7186e5 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:11:39 +05:30
Surendra Singh Lilhore
6a4297adc4 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 23:04:28 +05:30
Ayush Saxena
83ab853733 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:30:13 +05:30
Akira Ajisaka
1409c9da7c
HADOOP-16849. start-build-env.sh behaves incorrectly when username is numeric only. Contributed by Jihyun Cho. 
(cherry picked from commit 9709afe67d8ed45c3dfb53e45fe1efdc0814ac6c)

Conflicts:
	start-build-env.sh
 2020-02-12 14:11:00 +09:00
Kihwal Lee
bf09bfa2db HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)
(cherry picked from commit f1840669cfc92c033521bf95989479523e2a649d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
 2020-02-11 14:57:49 -06:00