Commit Graph

9431 Commits

Author SHA1 Message Date
Neil 5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 2021-11-16 18:21:50 +08:00
litao c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643)
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-16 11:19:14 +09:00
Symious 89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster (#3638) 2021-11-16 09:47:25 +08:00
Viraj Jasani e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648)
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-11-15 22:57:24 +09:00
GuoPhilipse 6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-14 19:59:15 +05:30
huhaiyang db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 2021-11-12 10:07:34 +08:00
jianghuazhu 3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. (#3629) 2021-11-12 09:52:02 +08:00
litao 1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636)
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-12 10:23:25 +09:00
litao 2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637)
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-10 19:22:32 +09:00
litao ea65fc26d8
HDFS-16298. Improve error msg for BlockMissingException (#3615)
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-10 19:16:45 +09:00
jianghuazhu b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 2021-11-10 11:32:52 +09:00
litao cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616)
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-09 17:06:28 +09:00
GuoPhilipse 9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability 2021-11-08 10:21:39 -08:00
Gautham B A 0a952e4d1f
HDFS-16304. Locate OpenSSL libs for libhdfspp (#3625) 2021-11-08 08:58:07 -08:00
Symious d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621) 2021-11-08 09:52:16 +08:00
Xiangyi Zhu e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553)
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-08 09:30:53 +09:00
Symious 6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613) 2021-11-05 11:03:07 +08:00
Gautham B A 1032724aa3
HDFS-16300. Use libcrypto in Windows for libhdfspp (#3617) 2021-11-04 09:19:11 -07:00
jianghuazhu 72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 2021-11-04 16:43:25 +09:00
jianghuazhu 18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. (#3602) 2021-11-04 09:51:11 +08:00
litao 359b03c83e
HDFS-16266. Add remote port information to HDFS audit log (#3538)
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-04 10:16:03 +09:00
daimin a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 2021-11-03 20:19:56 +00:00
Stephen O'Donnell 2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 2021-11-02 11:14:48 +00:00
jianghuazhu 618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544)
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-02 00:55:50 +09:00
jianghuazhu 0a68413a4f
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600)
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600). Contributed by JiangHua Zhu.
2021-10-31 08:48:52 +05:30
Masatake Iwasaki 634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. (#3374) 2021-10-31 09:29:34 +09:00
litao 6aa664c624
HDFS-16279. Print detail datanode info when process first storage report (#3564) 2021-10-29 10:27:00 +09:00
litao c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout (#3574) 2021-10-26 14:54:11 +08:00
Ahmed Hussein 62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566) 2021-10-23 16:13:07 -05:00
daimin 6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:12:13 +05:30
Ayush Saxena f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin."
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc.
2021-10-22 21:10:38 +05:30
daimin 7b913bc6bc
Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:09:37 +05:30
litao 2c37bebac4
HDFS-16280. Fix typo for ShortCircuitReplica#isStale (#3568). Contributed by tomscut.
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 08:21:30 +05:30
GuoPhilipse 1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 07:59:27 +05:30
Michael Kuchenbecker a73ff6915a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 2021-10-21 13:41:46 -07:00
Gautham B A a2f089df4d
HDFS-16278. Make HDFS snapshot tools cross platform (#3563) 2021-10-21 10:24:33 -07:00
Viraj Jasani 516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
daimin 5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 2021-10-18 11:55:55 +01:00
Xiangyi Zhu d09b380dc2
HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md (#3557)
Co-authored-by: zhuxiangyi <43412979@qq.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 13:53:41 +09:00
Ayush Saxena 263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:18 +05:30
litao d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt (#3554)
* HDFS-16274. Improve log for FSNamesystem#startFileInt

* fix failed unit test
2021-10-15 22:41:38 +05:30
Viraj Jasani 1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
LeonGao 7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 2021-10-13 18:14:03 -07:00
Gautham B A 107fe227eb
HDFS-16267. Make hdfs_df tool cross platform (#3542) 2021-10-13 09:11:21 -07:00
jianghuazhu ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). (#3497) 2021-10-13 17:21:03 +08:00
Gautham B A a05eb23a85
HDFS-16260. Make hdfs_deleteSnapshot tool cross platform (#3532) 2021-10-11 18:25:28 -07:00
jianghuazhu d7c01d083c
HDFS-16264.When adding block keys, the records come from the specific Block Pool. (#3533) 2021-10-11 19:49:58 +08:00
langlaile1221 88d8c3d0c8
HDFS-16241. Standby close reconstruction thread (#3493)
Co-authored-by: zhanghuazong <Zhz@20068867>
2021-10-11 16:11:31 +08:00
Viraj Jasani 3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules (#3530)
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-10-11 15:32:49 +09:00
Gautham B A 4f3dfb7c1c
HDFS-16265. Refactor HDFS tool tests for better reuse (#3536) 2021-10-09 11:00:43 -07:00