Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,805 Commits 326 Branches 371 Tags
Commit Graph

19805 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
S O'Donnell
4e1021975b HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:47:57 +00:00
Eric E Payne
80e394e81b YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit ede05b19d1723147430fc426161326d46698507f)
 2020-03-10 15:25:10 +00:00
Brahma Reddy Battula
e829631eed HADOOP-16871. Upgrade Netty version to 4.1.45.Final to handle CVE-2019-20444,CVE-2019-16869 
(cherry picked from commit c6b8a3038646697b77f6db54a2ef6266a9fc7888)
(cherry picked from commit 74fa55afc3d333d7d99397754f45b3fc54bc0f4a)

 Conflicts:
	hadoop-project/pom.xml
 2020-03-09 18:22:18 -07:00
Wei-Chiu Chuang
7bb5bf0e4e HADOOP-16643. Update netty4 to the latest 4.1.42. Contributed by Lisheng Sun. 
(cherry picked from commit 85af77c75768416db24ca506fd1704ce664ca92f)
 2020-03-09 17:58:15 -07:00
Weiwei Yang
79e5633867 HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 
(cherry picked from commit 6dfe00c71eb3721e9be3fc42349a81c4b013ada1)
 2020-03-08 22:01:59 -07:00
John Zhuge
c3249fa1ea
HADOOP-14206. TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature. Contributed by Jim Brennan. 
(cherry picked from commit 999096d82e6cc24023a77a69509af6820913f942)
 2020-03-08 00:02:02 -08:00
Siyao Meng
57783e7e24 HADOOP-16891. Upgrade jackson-databind to 2.9.10.3 (#1865) 
(cherry picked from commit e36b27260845c2eeb2211d01235cc6d3578b1942)
(cherry picked from commit 8a55178f4983f95e207656a6f81142f510c0cd7a)
 2020-02-28 17:03:38 -08:00
Wei-Chiu Chuang
58b025c8f4 HADOOP-16882. Update jackson-databind to 2.9.10.2 in branch-3.1, branch-2.10. Contributed by Lisheng Sun. 2020-02-28 16:24:38 -08:00
Surendra Singh Lilhore
64eb5b2709 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:47:54 +05:30
Mukund Thakur
ce6614565e HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running 
This uses the length of the file known at the start of the copy to determine the amount of data to copy.

* If a file is appended to during the copy, the original bytes are copied.
* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,
  distcp will now fail. Until now these failures were not detected.

Contributed by Mukund Thakur.

Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884
(cherry picked from commit 51c64b357d4bd1a0038e61df3d4b8ea0a3ad7449)
 2020-02-27 16:37:36 -08:00
Ayush Saxena
354e63b366 HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 
(cherry picked from commit df622cf4a32ee172ded6c4b3b97a1e49befc4f10)
 2020-02-27 16:37:30 -08:00
Masatake Iwasaki
abee7402e4 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 037ec8cfb1406ea3a8225a1b6306c2e78440353b)
 2020-02-27 16:37:23 -08:00
Konstantin V Shvachko
4af188587f HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ecbcb058b8bc0fbc3903acb56814c6d9608bc396)
 2020-02-27 16:02:30 -08:00
Ayush Saxena
7bfa123d58 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 
(cherry picked from commit b182dd809cf42aed66806901631ad6062ca0820c)
 2020-02-27 14:55:50 -08:00
Inigo Goiri
4924622e6e YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 
(cherry picked from commit a43510e21d01e6c78e98e7ad9469cbea70a66466)
 2020-02-27 13:20:17 -08:00
Istvan Fajth
b5022b0515 HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 
(cherry picked from commit 57aa048516f5c5fe02441d213b52ce1bbeddf823)
(cherry picked from commit e42ac486e7eecb6a24ac95f1ceaf61d24060adef)
 2020-02-27 07:49:45 -08:00
Kihwal Lee
b92477c638 HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed 
by Ahmed Hussein.

(cherry picked from commit 27cfda708ef66dfbe5f52a5f1e716298a294f3f7)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
 2020-02-27 09:45:12 -06:00
Elixir Kook
9ccefe2262
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 
(cherry picked from commit d608e94f9208354897a59dc8e533535b1b8a30b9)
 2020-02-26 17:32:47 +09:00
Stephen O'Donnell
8aaa8d1b71 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b9907b590ed2d73a0d68f079c7f4d754b1)
(cherry picked from commit 2377649cdb1307a72a946b95a9a04f316c6483ef)
 2020-02-25 13:29:15 -08:00
Takanobu Asanuma
241c5a14c5 HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li. 
(cherry picked from commit 5cbc4c54611f062690c3bfcf78d09d37c78ffd12)
 2020-02-25 11:10:22 +09:00
Wei-Chiu Chuang
268e35d3ae HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
(cherry picked from commit f9437b1817e09c5a3ee7b378a0f4faee9b853315)
 2020-02-24 13:37:59 -08:00
Ayush Saxena
d318b48e12 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:52 +05:30
Konstantin V Shvachko
cd0525054d HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 6f84269bcd5cdb08ca68b2d8276f66d34a2a7a0d)
 2020-02-21 10:56:31 -08:00
Masatake Iwasaki
8095e3b149 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java

(cherry picked from commit 610805ec7245769aebb36e52725522c42cb3dd88)
 2020-02-21 13:51:26 +09:00
Akira Ajisaka
3f8ac06cef
HADOOP-16869. Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure (#1855) 
(cherry picked from commit 7f35676f90f0730b8c9844cf00ee5a943f80d48d)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit b52f2012858d99c8959e63034dec947454363674)
 2020-02-21 12:24:41 +09:00
Wei-Chiu Chuang
b28347d282 HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze. 
(cherry picked from commit cb3f3cca01d6ab9f4befb1bcd31f384c37c0231a)
(cherry picked from commit d411a849e10910d24cf1bbc8d64405470e00f510)
 2020-02-18 21:54:00 -08:00
Sunil G
a6124cd2b8 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 
(cherry picked from commit 6526f95bd281fc011f8776d21ff933087c5924de)
 2020-02-19 11:18:53 +05:30
Kihwal Lee
c54051def5 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71cba068e7abf1c76ad629bfeec10852)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-17 16:07:03 -06:00
xiaoxiaopan118
b59fa3e571
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 
(cherry picked from commit 439d935e1df601ed998521443fbe6752040e7a84)
 2020-02-17 15:58:35 +09:00
Ayush Saxena
2190fb1fcf HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-16 10:09:40 +05:30
Ayush Saxena
cf5a7186e5 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:11:39 +05:30
Surendra Singh Lilhore
6a4297adc4 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 23:04:28 +05:30
Ayush Saxena
83ab853733 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:30:13 +05:30
Akira Ajisaka
1409c9da7c
HADOOP-16849. start-build-env.sh behaves incorrectly when username is numeric only. Contributed by Jihyun Cho. 
(cherry picked from commit 9709afe67d8ed45c3dfb53e45fe1efdc0814ac6c)

Conflicts:
	start-build-env.sh
 2020-02-12 14:11:00 +09:00
Kihwal Lee
bf09bfa2db HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)
(cherry picked from commit f1840669cfc92c033521bf95989479523e2a649d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
 2020-02-11 14:57:49 -06:00
Prabhu Joseph
00fde836f7 YARN-10022. RM Rest API to validate the CapacityScheduler Configuration change. 
Contributed by Kinga Marton.

(cherry-picked from commit 1ab9c692fa9107f10c0d8fbfbb9e74f54bddc960)
 2020-02-11 22:23:07 +05:30
Ayush Saxena
547f1798aa HDFS-15157. Fix compilation failure for branch-3.1. Contributed by Ayush Saxena 2020-02-09 08:14:33 +05:30
Jonathan Hung
5dfd1dcfe3 YARN-10116. Expose diagnostics in RMAppManager summary 
(cherry picked from commit 314e2f9d2e888fae1e5bf669aeeead84a928d282)
(cherry picked from commit 147897da4b420b4749f3c7b410f4c329632c3352)
 2020-02-05 11:17:24 -08:00
Kihwal Lee
a628ab0863 HADOOP-15787. TestIPC.testRTEDuringConnectionSetup fails with jdk8u242. Contributed by Zsolt Venczel. 
(cherry picked from commit a463cf75a0ab1f0dbb8cfa16c39a4e698bc1a625)
(cherry picked from commit 0b19c8e0860bf558d995951ab9ac4cd8a1674759)
 2020-02-05 13:16:34 -06:00
Chen Liang
466f77a5cf HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 
(cherry picked from commit 226cd3977bfc5948afff6a2d38a41e341089d5fe)
 2020-02-04 12:36:46 -08:00
Kihwal Lee
a55a0a1f6d HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 
(cherry picked from commit 10a60fbe20bb08cdd71076ea9bf2ebb3a2f6226e)
 2020-02-04 12:24:58 -06:00
Stephen O'Donnell
8ea4787f9f HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d931676b191cb4813ed1a283ebb24d4eb)
(cherry picked from commit e89436b1839aa87b4bedfc7e2d558c7541e73540)
 2020-01-31 16:13:53 -08:00
Inigo Goiri
ffaf0d3c82 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f60cbb8592d446694ec3d7710cda5142)
 2020-01-31 11:25:36 -08:00
Eric Badger
4af7d14ce2 YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 22:45:02 +00:00
Chen Liang
5e1d89b74d [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-29 11:15:35 -08:00
Eric E Payne
d4ecb65629 MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit b897f6834ba69d443c3acd1fab52261c00d675a1)
 2020-01-29 17:21:40 +00:00
Kihwal Lee
b8d87cfc32 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.

(cherry picked from commit 799d4c1cf4e8fe78eb9ab607a0449cdd075041fb)
 2020-01-29 11:01:51 -06:00
Ayush Saxena
1293199532 HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:17:21 +05:30
Ahmed Hussein
f3ff61865a MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 08251538fe2550d9dd86f9daf79994f5b8bdf7fa)
 2020-01-28 11:04:25 -06:00
Ahmed Hussein
1f315c28d9 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 3f01c481060585ccd37be9db8aa4d1e33d2e2d6b)
 2020-01-27 16:23:06 -06:00