Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,807 Commits 326 Branches 371 Tags
Commit Graph

6970 Commits

Author SHA1 Message Date
Akira Ajisaka
2ae21d5387
Addendum: Fix potential FSImage corruption. Contributed by Xiao Chen. 
(cherry picked from commit dd70b1f53836961029767660398819a3b04eec58)
 2018-10-31 17:03:47 +09:00
Xiao Chen
20c9a12bc7 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 
(cherry picked from commit db7e636824a36b90ba1c8e9b2fba1162771700fe)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java

(cherry picked from commit 399645ebc162371a63eb81840e01ed52261f43bc)
 2018-10-29 19:14:12 -07:00
Wei-Chiu Chuang
0f34ff7722 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fa01a8fa0e345c533e843504743baa00020be706.
 2018-10-28 17:07:29 -07:00
Anu Engineer
c5b704a34c HDFS-14028. HDFS OIV temporary dir deletes folder. 
Contributed by Adam Antal.

(cherry picked from commit 4f10d7e23fc0b757a9e94bc448187a6211b90f10)
 2018-10-26 16:55:55 -07:00
Arpit Agarwal
f6469adbb1 HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 
(cherry picked from commit 9b899f1ebd5126a756fceb43459164165488b203)
 2018-10-25 15:14:39 -07:00
Wei-Chiu Chuang
fa01a8fa0e HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fce019130e10964a644b94cddbab1c06)
(cherry picked from commit fd02c501c0cea3ec55956e11b390111519cedc00)
(cherry picked from commit ef9f8ca13d8dca8031c045bad801fceb3e00aa15)
 2018-10-25 09:38:26 -07:00
Ajay Kumar
2caf69debd HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Wei-Chiu Chuang. 2018-10-24 22:35:06 -07:00
Inigo Goiri
1d5390679e HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin. 
(cherry picked from commit 0e56c883cd2310f3ff9d62afb306b1ab27419c36)
 2018-10-24 16:14:22 -07:00
Xiao Chen
e8832418bf Fix potential FSImage corruption. Contributed by Daryn Sharp and Vinayakumar B. 
(cherry picked from commit f1996ccbaee734d423caa9d47a571cfff98ef42c)
(cherry picked from commit 96cedb87b94c07c11152580bf36978186d622b50)
 2018-10-24 16:10:51 -07:00
Inigo Goiri
6aef514795 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 
(cherry picked from commit c1874046e2f9275ba330090fcf12c5611c6d6fc4)
 2018-10-24 09:38:48 -07:00
Wei-Chiu Chuang
e4464f9598 Revert "HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar." 
This reverts commit d8cd07d5f5cc321d2f6c625a902f78e0c43433a5.
 2018-10-23 05:58:10 -07:00
Arpit Agarwal
d8cd07d5f5 HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 
(cherry picked from commit c58811c77d5c0442c404a5b2876e09eaf6d16073)
 2018-10-22 14:44:42 -07:00
drankye
b34c650a41 HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 
(cherry picked from commit 975d60685eaf9961bdbd3547600b3e38bb088835)
 2018-10-18 08:31:12 -07:00
Inigo Goiri
ea19a36132 HDFS-14000. RBF: Documentation should reflect right scripts for v3.0 and above. Contributed by CR Hota. 
(cherry picked from commit 7347fa2df3cfd48857f4ded3539a60101d026daf)
 2018-10-17 16:43:05 -07:00
Xiao Chen
b9f65d717c HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a0250ac3f2341ebe8a14d19b073e6224c8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2018-10-15 11:00:39 -07:00
Xiao Chen
ff7ca472d2 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
 2018-10-12 12:08:38 -07:00
Inigo Goiri
53b522af6d HDFS-13802. RBF: Remove FSCK from Router Web UI. Contributed by Fei Hui. 
(cherry picked from commit f63ee083db7c261207b726f0f44be2e5a92124f3)
 2018-10-12 10:12:07 -07:00
Hrishikesh Gadre
b170de8be5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 323b76bccfa153ef5ba52dc14876283d05618739)
 2018-10-09 19:54:56 -07:00
Inigo Goiri
8a9f61be4e HDFS-13964. RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice. Contributed by Ayush Saxena. 
(cherry picked from commit 3bfff0f79e7246cbb2989c720c14cfa2566a9d1a)
 2018-10-06 10:42:51 -07:00
Brahma Reddy Battula
b28dacf60c HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 
(cherry picked from commit e0ff8e2c10f38298b81624df1e62743ac171f18d)
 2018-09-26 23:55:02 +05:30
Siyao Meng
2449795b8e HDFS-13830. Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list. Contributed by Siyao Meng, Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-09-21 12:41:03 -07:00
Chen Liang
78e0c0a1bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:51:23 -07:00
Arpit Agarwal
ffef819ee8 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b1511f8f46866abe4eeec820f4191cc295)
 2018-09-19 13:49:57 -07:00
Inigo Goiri
3fc3a5c01c HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 
(cherry picked from commit 7f9a89e1b54a9712af50ffef70bed7cfb91ed34e)
 2018-09-19 12:59:42 -07:00
Inigo Goiri
b93aeb43a4 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0f49772d001c1fdb385110c705a622a)
 2018-09-19 09:31:58 -07:00
Wei-Chiu Chuang
823b1fceee HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 
(cherry picked from commit 28ceb34a725cd06d28fb51361c49bb45464f5368)
(cherry picked from commit 7442b7941dc57ebf4bd0ded4d46cfbf5af6fea81)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2018-09-19 03:20:26 -07:00
Wei-Chiu Chuang
3a32e84d3e HDFS-13886. HttpFSFileSystem.getFileStatus() doesn't return "snapshot enabled" bit. Contributed by Siyao Meng. 
(cherry picked from commit 44857476fa993fbf9c97f979b91e19d27632c10a)
(cherry picked from commit 595ce945704f37a4df5cb8c9b2c509b8d6a14ec4)
 2018-09-18 15:48:05 -07:00
Wei-Chiu Chuang
2ef3911f7c HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit a1de8cbac5fb9af403db2a02814575f0940d5f39)
(cherry picked from commit 829399a9f38db40b5bb247cb9c0d5f930c094e37)
 2018-09-18 15:48:04 -07:00
Inigo Goiri
f0537b5161 HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 
(cherry picked from commit a9c1c55cafc0338bb35d2991f217d4157463f1d3)
 2018-09-17 14:39:31 -07:00
Inigo Goiri
1314dc2397 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 
(cherry picked from commit e1b242a9844d6438c93cdc2ab2443d042b4d5aea)
 2018-09-13 09:18:11 -07:00
Brahma Reddy Battula
0c8aab690c HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang. 
(cherry picked from commit b2432d254c486d0e360b76b7b094a71a011678ab)
 2018-09-12 06:35:29 +05:30
Xiao Chen
a5da4df73c HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 
(cherry picked from commit 8e54da1511e78477c1d4655d5ff0a69d0330869f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java

(cherry picked from commit 2dd27c999b22c550058de0e6eca7209b346cd143)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
 2018-09-10 23:00:41 -07:00
Inigo Goiri
a9392bdb17 HDFS-13836. RBF: Handle mount table znode with null value. Contributed by yanghuafeng. 
(cherry picked from commit 527288ef891dc26019d003bd85ddfd50eb4f3b7b)
 2018-09-06 16:49:14 -07:00
Wei-Chiu Chuang
e99ed05444 HADOOP-15696. KMS performance regression due to too many open file descriptors after Jetty migration. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e780556ae9229fe7a90817eb4e5449d7eed35dd8)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java

(cherry picked from commit d1ea7df43d6a5623b139acbbc9992831d5c7e484)
 2018-09-05 10:18:56 -07:00
Hrishikesh Gadre
25337c045a HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 6ccb809c2d38a45e716153ba16e135cb76167b2b)
 2018-09-04 21:54:56 -07:00
Inigo Goiri
54d3189cfa HDFS-13857. RBF: Choose to enable the default nameservice to read/write files. Contributed by yanghuafeng. 
(cherry picked from commit 54f2044595206455484284b43e5976c8a1982aaf)
 2018-09-04 12:18:08 -07:00
Wei-Chiu Chuang
d06717ef94 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit d89a485b2545c759a0f62810ff9757ede6d622f2.
 2018-08-30 11:59:39 -07:00
Vinayakumar B
67a5491962 HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B. 
(cherry picked from commit 65977e5d8124be2bc208af25beed934933f170b3)
 2018-08-30 22:17:00 +05:30
Yiqun Lin
6a547856ef HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui. 
(cherry picked from commit 582cb10ec74ed5666946a3769002ceb80ba660cb)
 2018-08-30 11:24:25 +08:00
Brahma Reddy Battula
3b01279612 HDFS-13854. RBF: The ProcessingAvgTime and ProxyAvgTime should display by JMX with ms unit. Contributed by yanghuafeng. 
(cherry picked from commit 64ad0298d441559951bc9589a40f8aab17c93a5f)
 2018-08-29 08:32:05 +05:30
Wei-Chiu Chuang
72e1e1904b HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 2018-08-28 16:36:30 -07:00
Zsolt Venczel
2d1f819979 HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel. 
(cherry picked from commit 3e18b957ebdf20925224ab9c28e6c2f4b6bbdb24)
 2018-08-28 15:14:57 -07:00
Wei-Chiu Chuang
d89a485b25 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit 26c2a97c566969f50eb8e8432009724c51152a98)
(cherry picked from commit 41732e7b28bb70f7ba56cf6131ac10c062f93fc3)
 2018-08-27 16:04:16 -07:00
Konstantin V Shvachko
27abdb390b HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko. 
(cherry picked from commit a4121c71c29d13866a605d9c0d013e5de9c147c3)
 2018-08-24 18:51:42 -07:00
Yiqun Lin
4ef3025964 HDFS-13821. RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache. Contributed by Fei Hui. 2018-08-22 13:01:13 +08:00
Wei-Chiu Chuang
bd46906bd8 HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 
(cherry picked from commit 865650052b07c8a20d51306202354ac770ed36d5)
 2018-08-21 15:44:08 -07:00
Yiqun Lin
5e2e68d4bb HDFS-13750. RBF: Router ID in RouterRpcClient is always null. Contributed by Takanobu Asanuma. 
(cherry picked from commit 01ff8178148790f7b0112058cf08d23d031b6868)
 2018-08-20 23:03:19 +08:00
Xiao Chen
185c8f2abc HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 
(cherry picked from commit c67b0650ea10896c6289703595faef0d262c00b3)
 2018-08-16 23:15:58 -07:00
Daniel Templeton
27508086de HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh 
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
(cherry picked from commit 8512e1a91be3e340d919c7cdc9c09dfb762a6a4e)
(cherry picked from commit fbedf89377e540fb10239a880fc2e01ef7021b93)
 2018-08-16 15:02:18 -07:00
Xiao Chen
5d16301ad9 HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi. 
(cherry picked from commit cede33997f7ab09fc046017508b680e282289ce3)
 2018-08-14 11:59:05 -07:00