Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,907 Commits 326 Branches 371 Tags
Commit Graph

6995 Commits

Author SHA1 Message Date
Akira Ajisaka
f824f4dccb
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f09502a4f07342dee457e47bb52b03441ae)
 2019-04-05 12:31:38 +09:00
Stephen O'Donnell
79714fdfab HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
(cherry picked from commit 388f445dde577999b2d81f809adcfca8f0958499)
(cherry picked from commit d9899015ebf8a27e9ac339d8a8b3c9d88bcbacb9)
 2019-04-04 04:06:57 -07:00
Wei-Chiu Chuang
cbac102566 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070a124234c77f16193ee925d32ca9a20)
(cherry picked from commit c8703dda0727e17d759d7ad27f0caee88103a530)
(cherry picked from commit 2a94603ae66d9000c0bb07df0d592279339af103)
 2019-04-03 11:02:14 -07:00
Gabor Bota
461df1b511
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8ecb83b666d4ed223fc270e1a46642c1)
 2019-04-01 13:53:11 +09:00
Takanobu Asanuma
abb06c9d7c HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb48ca26a629d4d5f3f07e2858d09594)
 2019-03-27 03:30:12 +09:00
Inigo Goiri
5a8445789b HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631cb8a0f67bbc39b5b5a1b0a81ca3b3c)
 2019-03-15 13:59:51 -07:00
Erik Krogen
9a5daa5ec6 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
(cherry picked from commit fec7c5f3ebbaea7b290e0904570c3a485b541a22)
(cherry picked from commit cb76fc5618c582887e4e71b98f3065c5a916334a)
 2019-03-13 13:21:40 -07:00
Erik Krogen
c924011f94 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3f74968c92751f0972746b012e72810)
(cherry-picked from commit 6d076dd5e8be8fee5062a04d4cd2aa0c5da0ea56)
(cherry-picked from commit b0b80002402cf5fbe819367d96e42098c5c359ea)
 2019-03-07 08:45:20 -08:00
Wei-Chiu Chuang
e23a448e0e HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587aa66ac99ec5f5b50827ec3e0a327613)
(cherry picked from commit e58ccca3ce131c955ceb115cd0b75e452eea828b)
(cherry picked from commit d951497f57cf6556b0916cad08576481dfe2ae06)
 2019-03-04 10:46:16 -08:00
Erik Krogen
b170626c74 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:58 -08:00
Kitti Nanasi
ad131757f2 HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 49ddd8a6ed5b40d12defb0771b4c8b53d4ffde3f)
(cherry picked from commit 38541b23753c5d0e2cb9baf1d2da536f846dc38c)
(cherry picked from commit c25b641cec8aca9c16759d5e392ffbc27a158a67)
 2019-02-05 15:15:53 -08:00
Masatake Iwasaki
00eee37a93 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7bdf437713a8c6ca3c535c4b6d6992e)
 2019-01-29 02:31:59 +09:00
Dinesh Chitlangia
36ff7a4d02 HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 45caeee6cfcf1ae3355cd880402159cf31e94a8a)
(cherry picked from commit eb77e60163e318ca42548e134fe79dba7289fdc6)
(cherry picked from commit b63941516f78e0d6e22ecc629fc82b16fd17ad87)
 2019-01-25 07:38:49 -08:00
Takanobu Asanuma
5b8a99bf49 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 
(cherry picked from commit 7d6792e5d2001e1a83ae75a4777324a3b01c557a)
 2019-01-23 10:06:30 +09:00
Wei-Chiu Chuang
054594c614 HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang. 
(cherry picked from commit e9a005dfcf4d2249235aeff4da85b321dddc1ca9)
(cherry picked from commit c48640ffbd3852b75f898325ae0b3ea0ba864578)
(cherry picked from commit f24684ca9cd70326e24376a33cda937c98295af0)
 2018-12-28 10:39:47 -08:00
Brahma Reddy Battula
6895550043 HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena. 
(cherry picked from commit f534736867eed962899615ca1b7eb68bcf591d17)
 2018-11-30 00:21:07 +05:30
Vinayakumar B
8569555fd1 HDFS-13816. dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException. Contributed by Vinayakumar B. 
(cherry picked from commit b09828145432c8d986ac8f05ec33608d8e611328)
 2018-11-26 18:23:20 +05:30
Vinayakumar B
a19c72d7e8 HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena. 
(cherry picked from commit 886b935630bf2d46f4c6773de013c093d2e44c38)
 2018-11-16 18:02:01 +05:30
Vinayakumar B
1b6fa47334 HDFS-13963. NN UI is broken with IE11. Contributed by Ayush Saxena. 
(cherry picked from commit 8571507efa3af3e4a5213a7e24c65e26162eac4d)
 2018-11-16 17:36:50 +05:30
Brahma Reddy Battula
04bba91585 HDFS-14070. Refactor NameNodeWebHdfsMethods to allow better extensibility. Contributed by CR Hota 
(cherry picked from commit e7b63baca1e10b28d8b4462fd80537b871951aa3)
 2018-11-13 13:12:56 +05:30
Arpit Agarwal
f514c08900 HDFS-14065. Failed Storage Locations shows nothing in the Datanode Volume Failures. Contributed by Ayush Saxena. 
(cherry picked from commit b6d4e19f34f474ea8068ebb374f55e0db2f714da)
 2018-11-12 15:31:56 -08:00
Kitti Nanasi
bbc94de477 HDFS-14039. ec -listPolicies doesn't show correct state for the default policy when the default is not RS(6,3). Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 8d99648c203004045a9339ad27258092969145d6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 01b8197893c84fa24bb94945f85b3e2ebe8259a5)
 2018-11-08 10:22:32 -08:00
Vinayakumar B
76be3515bf HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena 
(cherry picked from commit 8df2eb8119188b8e5515295523afc23046e1db81)
 2018-11-08 10:22:32 -08:00
Inigo Goiri
c4d390a775 HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 
(cherry picked from commit 887244de4adebe27693ed4ad3296a6f700cfa8c1)
 2018-11-06 11:21:19 -08:00
Inigo Goiri
4b47daa640 HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 
(cherry picked from commit f3296501e09fa7f1e81548dfcefa56f20fe337ca)
(cherry picked from commit 9bf4f3d61403b06f3d6a092dacab382c7c131e19)
 2018-11-05 16:53:28 -08:00
Akira Ajisaka
2ae21d5387
Addendum: Fix potential FSImage corruption. Contributed by Xiao Chen. 
(cherry picked from commit dd70b1f53836961029767660398819a3b04eec58)
 2018-10-31 17:03:47 +09:00
Xiao Chen
20c9a12bc7 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 
(cherry picked from commit db7e636824a36b90ba1c8e9b2fba1162771700fe)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java

(cherry picked from commit 399645ebc162371a63eb81840e01ed52261f43bc)
 2018-10-29 19:14:12 -07:00
Wei-Chiu Chuang
0f34ff7722 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fa01a8fa0e345c533e843504743baa00020be706.
 2018-10-28 17:07:29 -07:00
Anu Engineer
c5b704a34c HDFS-14028. HDFS OIV temporary dir deletes folder. 
Contributed by Adam Antal.

(cherry picked from commit 4f10d7e23fc0b757a9e94bc448187a6211b90f10)
 2018-10-26 16:55:55 -07:00
Arpit Agarwal
f6469adbb1 HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 
(cherry picked from commit 9b899f1ebd5126a756fceb43459164165488b203)
 2018-10-25 15:14:39 -07:00
Wei-Chiu Chuang
fa01a8fa0e HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fce019130e10964a644b94cddbab1c06)
(cherry picked from commit fd02c501c0cea3ec55956e11b390111519cedc00)
(cherry picked from commit ef9f8ca13d8dca8031c045bad801fceb3e00aa15)
 2018-10-25 09:38:26 -07:00
Ajay Kumar
2caf69debd HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Wei-Chiu Chuang. 2018-10-24 22:35:06 -07:00
Inigo Goiri
1d5390679e HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin. 
(cherry picked from commit 0e56c883cd2310f3ff9d62afb306b1ab27419c36)
 2018-10-24 16:14:22 -07:00
Xiao Chen
e8832418bf Fix potential FSImage corruption. Contributed by Daryn Sharp and Vinayakumar B. 
(cherry picked from commit f1996ccbaee734d423caa9d47a571cfff98ef42c)
(cherry picked from commit 96cedb87b94c07c11152580bf36978186d622b50)
 2018-10-24 16:10:51 -07:00
Inigo Goiri
6aef514795 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 
(cherry picked from commit c1874046e2f9275ba330090fcf12c5611c6d6fc4)
 2018-10-24 09:38:48 -07:00
Wei-Chiu Chuang
e4464f9598 Revert "HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar." 
This reverts commit d8cd07d5f5cc321d2f6c625a902f78e0c43433a5.
 2018-10-23 05:58:10 -07:00
Arpit Agarwal
d8cd07d5f5 HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 
(cherry picked from commit c58811c77d5c0442c404a5b2876e09eaf6d16073)
 2018-10-22 14:44:42 -07:00
drankye
b34c650a41 HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 
(cherry picked from commit 975d60685eaf9961bdbd3547600b3e38bb088835)
 2018-10-18 08:31:12 -07:00
Inigo Goiri
ea19a36132 HDFS-14000. RBF: Documentation should reflect right scripts for v3.0 and above. Contributed by CR Hota. 
(cherry picked from commit 7347fa2df3cfd48857f4ded3539a60101d026daf)
 2018-10-17 16:43:05 -07:00
Xiao Chen
b9f65d717c HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a0250ac3f2341ebe8a14d19b073e6224c8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2018-10-15 11:00:39 -07:00
Xiao Chen
ff7ca472d2 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
 2018-10-12 12:08:38 -07:00
Inigo Goiri
53b522af6d HDFS-13802. RBF: Remove FSCK from Router Web UI. Contributed by Fei Hui. 
(cherry picked from commit f63ee083db7c261207b726f0f44be2e5a92124f3)
 2018-10-12 10:12:07 -07:00
Hrishikesh Gadre
b170de8be5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 323b76bccfa153ef5ba52dc14876283d05618739)
 2018-10-09 19:54:56 -07:00
Inigo Goiri
8a9f61be4e HDFS-13964. RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice. Contributed by Ayush Saxena. 
(cherry picked from commit 3bfff0f79e7246cbb2989c720c14cfa2566a9d1a)
 2018-10-06 10:42:51 -07:00
Brahma Reddy Battula
b28dacf60c HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 
(cherry picked from commit e0ff8e2c10f38298b81624df1e62743ac171f18d)
 2018-09-26 23:55:02 +05:30
Siyao Meng
2449795b8e HDFS-13830. Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list. Contributed by Siyao Meng, Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-09-21 12:41:03 -07:00
Chen Liang
78e0c0a1bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:51:23 -07:00
Arpit Agarwal
ffef819ee8 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b1511f8f46866abe4eeec820f4191cc295)
 2018-09-19 13:49:57 -07:00
Inigo Goiri
3fc3a5c01c HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 
(cherry picked from commit 7f9a89e1b54a9712af50ffef70bed7cfb91ed34e)
 2018-09-19 12:59:42 -07:00
Inigo Goiri
b93aeb43a4 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0f49772d001c1fdb385110c705a622a)
 2018-09-19 09:31:58 -07:00