Commit Graph

5703 Commits

Author SHA1 Message Date
Kai Zheng b5af9be72c HDFS-8668. Erasure Coding: revisit buffer used for encoding and decoding. Contributed by Sammi Chen 2016-08-13 13:52:37 +08:00
Wei-Chiu Chuang a428d4f50e HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 2016-08-10 22:17:35 -07:00
Kai Zheng e7e8aed208 HDFS-10720. Fix intermittent test failure of TestDataNodeErasureCodingMetrics. Contributed by Rakesh R 2016-08-12 11:15:05 +08:00
Xiaoyu Yao ec289bbece HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:49:36 -07:00
Xiaoyu Yao fdafd0cb4f Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao."
This reverts commit 9f816d7ca7.
2016-08-10 15:47:55 -07:00
Xiaoyu Yao 9f816d7ca7 Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:14:38 -07:00
Yuanbo Liu 10e84c6a6e HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 2016-08-10 11:07:42 -07:00
Wei-Chiu Chuang d00d3add9e HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 2016-08-09 18:11:34 -07:00
Lei Xu 9c6a4383ca HDFS-10681. DiskBalancer: query command should report Plan file path apart from PlanID. (Manoj Govindassamy via lei) 2016-08-09 15:59:22 -07:00
Lei Xu cc48251bfd HDFS-10457. DataNode should not auto-format block pool directory if VERSION is missing. (Wei-Chiu Chuang via lei) 2016-08-09 15:51:19 -07:00
Kihwal Lee 0f701f433d HDFS-10738. Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure. Contributed by Rakesh R. 2016-08-09 14:54:11 -05:00
Kihwal Lee b10c936020 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 2016-08-09 11:56:21 -05:00
Arpit Agarwal 8c0638471f HADOOP-10682. Replace FsDatasetImpl object lock with a separate lock object. (Chen Liang) 2016-08-08 12:02:53 -07:00
Konstantin V Shvachko 4d3af47f27 HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 2016-08-07 14:29:26 -07:00
Akira Ajisaka ef432579a7 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 2016-08-05 17:31:11 +09:00
Wei-Chiu Chuang 18d9e6ec0b HDFS-10588. False alarm in datanode log - ERROR - Disk Balancer is not enabled. Contributed by Weiwei Yang. 2016-08-04 19:40:33 -07:00
Kihwal Lee 438a9f047e HDFS-10673. Optimize FSPermissionChecker's internal path usage. Contributed by Daryn Sharp. 2016-08-04 16:14:13 -05:00
Kihwal Lee 57369a678c HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 2016-08-04 14:55:21 -05:00
Kihwal Lee 4a8e6dc02f HDFS-10722. Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs. Contributed by Daryn Sharp. 2016-08-04 11:53:44 -05:00
Tsz-Wo Nicholas Sze 8c9b44eaff In Balancer, the target task should be removed when its size < 0. Contributed by Yiqun Lin 2016-08-04 09:45:40 -07:00
Kihwal Lee 6ae39199da HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp. 2016-08-04 09:07:34 -05:00
Kihwal Lee 6f63566694 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 2016-08-03 15:20:20 -05:00
Jing Zhao f4ba5ff1d7 HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 2016-08-03 12:42:05 -07:00
Kihwal Lee 22ef5286bc HDFS-10674. Optimize creating a full path from an inode. Contributed by Daryn Sharp. 2016-08-03 13:12:43 -05:00
Kihwal Lee 58db263e93 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 2016-08-03 12:14:59 -05:00
Kihwal Lee bebf10d245 HDFS-10656. Optimize conversion of byte arrays back to path string. Contributed by Daryn Sharp. 2016-08-03 11:53:41 -05:00
Kihwal Lee 5e5b8793fb HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on branch-2. Contributed by Eric Badger. 2016-08-02 15:37:36 -05:00
Akira Ajisaka a5fb298e56 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 2016-08-02 17:07:59 +09:00
Konstantin V Shvachko c4463f2ef2 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi."
This reverts commit 85a20508bd.
2016-08-01 22:34:40 -07:00
Daryn Sharp 9f473cf903 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 2016-08-01 10:14:28 -05:00
Yongjun Zhang 8d32bd884d HDFS-10703. HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) 2016-07-29 23:20:14 -07:00
Akira Ajisaka 204a2055b1 HDFS-10691. FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException. Contributed by Yiqun Lin. 2016-07-29 15:39:48 +09:00
Xiao Chen d9aae22fdf HDFS-9276. Failed to Update HDFS Delegation Token for long running application in HA mode. Contributed by Liangliang Gu and John Zhuge 2016-07-28 16:29:40 -07:00
Xiaoyu Yao ce3d68e9c3 HDFS-10676. Add namenode metric to measure time spent in generating EDEKs. Contributed by Hanisha Koneru. 2016-07-28 16:07:45 -07:00
Lei Xu a3d0cba810 HDFS-10689. Hdfs dfs chmod should reset sticky bit permission when the bit is omitted in the octal mode. (Manoj Govindassamy via Lei Xu) 2016-07-28 13:52:14 -07:00
Lei Xu 67406460f0 HDFS-4176. EditLogTailer should call rollEdits with a timeout. (Lei Xu) 2016-07-28 13:32:00 -07:00
Xiao Chen 328c855a57 HDFS-10650. DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission. Contributed by John Zhuge. 2016-07-28 13:15:02 -07:00
Akira Ajisaka 414fbfab41 HDFS-10696. TestHDFSCLI fails. Contributed by Kai Sasaki. 2016-07-28 16:58:23 +09:00
Andrew Wang 098ec2b11f HDFS-10519. Add a configuration option to enable in-progress edit log tailing. Contributed by Jiayi Zhou. 2016-07-27 17:55:41 -07:00
Yongjun Zhang eb7ff0c992 HDFS-10667. Report more accurate info about data corruption location. (Yuanbo Liu via Yongjun Zhang) 2016-07-27 13:58:04 -07:00
Akira Ajisaka 54fe17a607 HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák. 2016-07-27 19:41:09 +09:00
Brahma Reddy Battula 55d5993a8e HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 2016-07-27 10:26:42 +05:30
Wei-Chiu Chuang d84ab8a578 HDFS-10598. DiskBalancer does not execute multi-steps plan. Contributed by Lei (Eddy) Xu. 2016-07-26 16:01:50 -07:00
Arpit Agarwal d2cf8b54c5 HDFS-10642. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. (Contributed by Mingliang Liu) 2016-07-26 12:27:46 -07:00
Wei-Chiu Chuang 25f0a9b3ab HDFS-9937. Update dfsadmin command line help and HdfsQuotaAdminGuide. Contributed by Kai Sasaki. 2016-07-26 08:01:42 -07:00
Steve Loughran 2aa18e1925 Revert "HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu."
This reverts commit 7cac7655fd.
2016-07-26 14:30:02 +01:00
Brahma Reddy Battula 7cac7655fd HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 2016-07-26 10:54:24 +05:30
Vinitha Reddy Gankidi 85a20508bd HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi. 2016-07-25 18:50:59 -07:00
Masatake Iwasaki 59466b8c18 HDFS-10671. Fix typo in HdfsRollingUpgrade.md. Contributed by Yiqun Lin. 2016-07-26 10:48:21 +09:00
Jing Zhao 0cde9e12a7 HDFS-10688. BPServiceActor may run into a tight loop for sending block report when hitting IOException. Contributed by Chen Liang. 2016-07-25 18:41:13 -07:00
Kai Zheng b79ba4fe96 HDFS-10651. Clean up some configuration related codes about legacy block reader. Contributed by Youwei Wang 2016-07-23 17:45:33 +08:00
Arpit Agarwal c6e3a0020f HDFS-10660. Expose storage policy apis via HDFSAdmin interface. (Contributed by Rakesh R) 2016-07-22 12:33:50 -07:00
Uma Maheswara Rao G 77ac04efe4 HDFS-10565: Erasure Coding: Document about the current allowed storage policies for EC Striped mode files 2016-07-22 11:08:20 -07:00
Xiao Chen 132deb4cac HDFS-10225. DataNode hot swap drives should disallow storage type changes. Contributed by Lei (Eddy) Xu. 2016-07-21 16:41:02 -07:00
Akira Ajisaka fcde6940e0 HDFS-10287. MiniDFSCluster should implement AutoCloseable. Contributed by Andras Bokor. 2016-07-21 11:26:08 -07:00
Jing Zhao bd3dcf46e2 HDFS-10653. Optimize conversion from path string to components. Contributed by Daryn Sharp. 2016-07-21 11:14:39 -07:00
Akira Ajisaka 38128baff4 HDFS-10425. Clean up NNStorage and TestSaveNamespace. Contributed by Andras Bokor. 2016-07-20 10:44:11 -07:00
Akira Ajisaka f7dabe3add HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg. 2016-07-19 15:13:01 -07:00
Akira Ajisaka fc570b55b9 HDFS-10603. Fix flaky tests in org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot. Contributed by Yiqun Lin. 2016-07-19 13:49:24 -07:00
Akira Ajisaka 5b4a708704 HDFS-10639. Fix typos in HDFSDiskbalancer.md. Contributed by Yiqun Lin. 2016-07-15 21:41:57 -07:00
Andrew Wang da456ffd62 Preparing for 3.0.0-alpha2 development 2016-07-15 19:04:17 -07:00
Akira Ajisaka c48e9d608c HDFS-10628. Log HDFS Balancer exit message to its own log. Contributed by Jiayi Zhou. 2016-07-15 11:42:44 -07:00
Akira Ajisaka 5537c6b234 HADOOP-13351. TestDFSClientSocketSize buffer size tests are flaky. Contributed by Aaron Fabbri and Mingliang Liu. 2016-07-14 14:40:58 -07:00
Lei Xu 382dff7475 HDFS-10600. PlanCommand#getThrsholdPercentage should not use throughput value. (Yiqun Lin via lei) 2016-07-14 10:40:34 -07:00
Kihwal Lee 2bbc3ea1b5 HDFS-10617. PendingReconstructionBlocks.size() should be synchronized. Contributed by Eric Badger. 2016-07-13 21:59:35 -05:00
Uma Maheswara Rao G 438b7c5935 HDFS-10590: Fix TestReconstructStripedBlocks.testCountLiveReplicas test failures. Contributed by Rakesh R 2016-07-13 00:07:11 -07:00
Zhe Zhang 087290e6b1 HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. 2016-07-12 23:18:37 -07:00
Jitendra Pandey c447efebdb HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS protection. Contributed by Anu Engineer. 2016-07-11 14:55:33 -07:00
Yongjun Zhang da6f1b88dd HDFS-10512. VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks. Contributed by Wei-Chiu Chuang and Yiqun Lin. 2016-07-08 19:40:44 -07:00
Akira Ajisaka 932aed64d7 HDFS-10336. TestBalancer failing intermittently because of not reseting UserGroupInformation completely. Contributed by Yiqun Lin. 2016-07-08 14:39:11 -07:00
Jitendra Pandey 6e597600f7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 14:58:19 -07:00
Uma Maheswara Rao G ab092c56c2 HDFS-10592:Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning. Contributed by Rakesh R 2016-07-07 13:52:35 -07:00
Andrew Wang b8f93cd275 HDFS-10488. Update WebHDFS documentation regarding CREATE and MKDIR default permissions. Contributed by Wellington Chevreuil. 2016-07-07 10:36:55 -07:00
Akira Ajisaka a3f93beaa1 HDFS-10582. Change deprecated configuration fs.checkpoint.dir to dfs.namenode.checkpoint.dir in HDFS Commands Doc. Contributed by Pan Yuxuan. 2016-07-06 17:51:42 -07:00
Akira Ajisaka 8b4b5259d5 HDFS-10589. Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params. Contributed by shenyinjie. 2016-07-04 07:15:04 -07:00
Akira Ajisaka 2e835d986e HDFS-10572. Fix TestOfflineEditsViewer#testGenerated. Contributed by Surendra Singh Lilhore. 2016-07-03 21:11:56 -07:00
Kai Zheng 8b281bce85 HDFS-10548. Remove the long deprecated BlockReaderRemote. Contributed by Kai Zheng 2016-07-03 11:56:23 +08:00
Colin Patrick Mccabe e4a2545620 HDFS-9805. Add server-side configuration for enabling TCP_NODELAY for DataTransferProtocol and default it to true (Gary Helmling via cmccabe) 2016-06-29 12:41:42 -07:00
Andrew Wang c0829f4493 HDFS-6434. Default permission for creating file should be 644 for WebHdfs/HttpFS. Contributed by Wellington Chevreuil. 2016-06-28 13:55:26 -07:00
Allen Wittenauer 23c3ff85a9 HADOOP-13209. replace slaves with workers (John Smith via aw) 2016-06-28 05:53:03 -07:00
Vinayakumar B 2a0082c51d HDFS-10440. Improve DataNode web UI (Contributed by Weiwei Yang) 2016-06-28 16:49:39 +05:30
Vinayakumar B 73615a789d HDFS-10536. Standby NN can not trigger log roll after EditLogTailer thread failed 3 times in EditLogTailer.triggerActiveLogRoll method (Contributed by XingFeng Shen 2016-06-26 15:45:32 +05:30
Sangjin Lee bf74dbf80d HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee) 2016-06-24 14:44:15 -07:00
Zhe Zhang 22be604fa3 Revert "HDFS-10534. NameNode WebUI should display DataNode usage rate with a certain percentile. Contributed by Kai Sasaki."
This reverts commit 0424056a77.
2016-06-24 09:36:03 -07:00
Zhe Zhang 0424056a77 HDFS-10534. NameNode WebUI should display DataNode usage rate with a certain percentile. Contributed by Kai Sasaki. 2016-06-24 08:58:29 -07:00
Anu Engineer b2584bee45 HDFS-10571. TestDiskBalancerCommand#testPlanNode failed with IllegalArgumentException. Contributed by Xiaobing Zhou. 2016-06-23 18:27:02 -07:00
Anu Engineer 8a6e354122 HDFS-10552. DiskBalancer "-query" results in NPE if no plan for the node. Contributed by Anu Engineer. 2016-06-23 18:27:02 -07:00
Anu Engineer e8de28181a HDFS-10551. o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected. Contributed by Anu Engineer. 2016-06-23 18:27:02 -07:00
Arpit Agarwal 66fa34c839 HDFS-10557. Fix handling of the -fs Generic option. (Arpit Agarwal) 2016-06-23 18:27:01 -07:00
Anu Engineer 7b23ad1ef7 HDFS-10550. DiskBalancer: fix issue of order dependency in iteration in ReportCommand test. Contributed by Xiaobing Zhou. 2016-06-23 18:27:01 -07:00
Anu Engineer c6ed54808d HDFS-10547. DiskBalancer: fix whitespace issue in doc files. Contributed by Anu Engineer. 2016-06-23 18:27:01 -07:00
Anu Engineer 0774412e41 HDFS-10545. DiskBalancer: PlanCommand should use -fs instead of -uri to be consistent with other hdfs commands. Contributed by Anu Engineer. 2016-06-23 18:27:01 -07:00
Anu Engineer 5b8e1c26d7 HDFS-10541. Diskbalancer: When no actions in plan, error message says "Plan was generated more than 24 hours ago". Contributed by Anu Engineer. 2016-06-23 18:27:00 -07:00
Anu Engineer cb68e5b3bd HDFS-10540. Diskbalancer: The CLI error message for disk balancer is not enabled is not clear. Contributed by Anu Engineer. 2016-06-23 18:27:00 -07:00
Anu Engineer 3225c24e0e HDFS-10520. DiskBalancer: Fix Checkstyle issues in test code. Contributed by Anu Engineer. 2016-06-23 18:27:00 -07:00
Anu Engineer 7e2be5c4a0 HDFS-10518. DiskBalancer: Pretty-print json in Query command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer af11ab34d0 HDFS-10517. DiskBalancer: Support help command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer b502102bb1 HDFS-9461. DiskBalancer: Add Report Command. Contributed by Xiaobing Zhou. 2016-06-23 18:21:08 -07:00
Anu Engineer 121142cf95 HDFS-10501. DiskBalancer: Use the default datanode port if port is not provided. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer 78a1032b71 HDFS-10500. Diskbalancer: Print out information when a plan is not generated. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00