Commit Graph

6373 Commits

Author SHA1 Message Date
Anu Engineer 6b5dee9c38 HDFS-11492. Ozone: Add the ability to handle sendContainerReport Command. Contributed by Anu Engineer. 2017-03-21 10:43:27 -07:00
Yiqun Lin 3b908f71c5 HDFS-11358. DiskBalancer: Report command supports reading nodes from host file. Contributed by Yiqun Lin. 2017-03-21 17:44:17 +08:00
Arpit Agarwal e7c8da614c HDFS-11545. Propagate DataNode's slow disks info to the NameNode via Heartbeats. Contributed by Hanisha Koneru. 2017-03-20 21:54:58 -07:00
Chris Douglas b104f3a282 HDFS-6648. Order of namenodes in ConfiguredFailoverProxyProvider is undefined. Contributed by Inigo Goiri 2017-03-20 17:15:13 -07:00
Anu Engineer 603f2c18ec HDFS-11444. Ozone: Fix datanode ID handling in MiniOzoneCluster. Contributed by Weiwei Yang. 2017-03-20 13:18:26 -07:00
Arpit Agarwal ffa160ddb8 HDFS-11547. Add logs for slow BlockReceiver while writing data to disk. Contributed by Xiaobing Zhou. 2017-03-17 19:02:24 -07:00
Hanisha Koneru 7f8e928400 HDFS-11517. Expose slow disks via DataNode JMX. Contributed by Hanisha Koneru 2017-03-17 15:42:25 -07:00
Andrew Wang 92ea6d74ec HDFS-11516. Admin command line should print message to stderr in failure case. Contributed by Kai Sasaki. 2017-03-17 11:26:02 -07:00
Xiaobing Zhou 7536815039 HDFS-10394. move declaration of okhttp version from hdfs-client to hadoop-project POM. Contributed by Xiaobing Zhou. 2017-03-16 18:13:43 -07:00
Andrew Wang 4812518b23 HDFS-10530. BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy. Contributed by Manoj Govindassamy and Rui Gao. 2017-03-16 15:07:38 -07:00
Kihwal Lee ba62b50eba HDFS-10601. Improve log message to include hostname when the NameNode is in safemode. Contributed by Kuhu Shukla. 2017-03-16 10:54:59 -05:00
Hanisha Koneru d69a82c89c HDFS-11511. Support Timeout when checking single disk. Contributed by Hanisha Koneru. 2017-03-15 18:01:45 -07:00
Chen Liang 615ac09499 HDFS-11419. DFSTopologyNodeImpl#chooseRandom optimizations. Contributed by Chen Liang. 2017-03-15 12:28:22 -07:00
Yiqun Lin 4416a07c9c HDFS-11420. Edit file should not be processed by the same type processor in OfflineEditsViewer. Contributed by Yiqun Lin. 2017-03-15 18:05:03 +08:00
Xiaoyu Yao 65487b579e HDFS-11491. Ozone: SCM: Add close container RPC. Contributed by Anu Engineer. 2017-03-14 21:28:44 -07:00
Xiaoyu Yao 72f4e3f347 HDFS-11487. Ozone: Exclude container protobuf files from findbugs check(amends previous fix). Contributed by Yuanbo Liu. 2017-03-14 21:18:44 -07:00
Andrew Wang cc1292e73a HDFS-9705. Refine the behaviour of getFileChecksum when length = 0. Contributed by Kai Zheng and SammiChen. 2017-03-14 16:41:10 -07:00
Anu Engineer 39058dd601 HDFS-11469. Ozone: SCM: Container allocation based on node report. Contributed by Xiaoyu Yao. 2017-03-14 11:54:26 -07:00
Andrew Wang 7515e75103 HDFS-11505. Do not enable any erasure coding policies by default. Contributed by Manoj Govindassamy. 2017-03-14 11:47:25 -07:00
Yiqun Lin 023b941e3b HDFS-11526. Fix confusing block recovery message. Contributed by Yiqun Lin. 2017-03-14 17:49:48 +08:00
Chen Liang 9832ae0ed8 HDFS-11482. Add storage type demand to into DFSNetworkTopology#chooseRandom. Contributed by Chen Liang. 2017-03-13 17:30:10 -07:00
Jing Zhao 55796a0946 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar. 2017-03-13 14:24:51 -07:00
Yiqun Lin 79924266f8 HDFS-11512. Increase timeout on TestShortCircuitLocalRead#testSkipWithVerifyChecksum. Contributed by Eric Badger. 2017-03-13 18:22:30 +08:00
Xiaoyu Yao 8eca9824cf HDFS-11071. Ozone: SCM: Move SCM config keys to ScmConfig. Contributed by Weiwei Yang. 2017-03-10 16:24:24 -08:00
Lei Xu 6d356b6b4d HDFS-11340. DataNode reconfigure for disks doesn't remove the failed volumes. (Manoj Govindassamy via lei) 2017-03-10 14:37:13 -08:00
Andrew Wang 819808a016 HDFS-11506. Move ErasureCodingPolicyManager#getSystemDefaultPolicy to test code. Contributed by Manoj Govindassamy. 2017-03-09 17:29:11 -08:00
Tsz-Wo Nicholas Sze fbc8099b63 HDFS-11513. Ozone: Separate XceiverServer and XceiverClient into interfaces and implementations. 2017-03-09 11:03:20 -08:00
Masatake Iwasaki 385d2cb777 HDFS-11499. Decommissioning stuck because of failing recovery. Contributed by Lukas Majercak and Manoj Govindassamy. 2017-03-09 21:13:50 +09:00
Anu Engineer 5e95bdeef2 HDFS-10928. Ozone:SCM: Support MXBean for SCM and NodeManager. Contributed by Weiwei Yang. 2017-03-08 18:52:57 -08:00
Andrew Wang 33a38a5341 HDFS-11314. Enforce set of enabled EC policies on the NameNode. 2017-03-08 16:41:44 -08:00
Andrew Wang 5ca6ef0c26 HDFS-10983. OIV tool should make an EC file explicit. Contributed by Manoj Govindassamy. 2017-03-08 15:36:19 -08:00
Anu Engineer cc63937e31 merge trunk to HDFS-7240 2017-03-08 15:34:04 -08:00
Andrew Wang 6b6c61ee74 HDFS-11152. Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors. Contributed by SammiChen. 2017-03-08 15:34:04 -08:00
Arpit Agarwal 647de3c41e HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-08 15:34:01 -08:00
Arpit Agarwal 79b8d7ed38 HDFS-10838. Last full block report received time for each DN should be easily discoverable. Contributed by Surendra Singh Lilhore. 2017-03-08 15:33:59 -08:00
Andrew Wang 52a53fe71f HDFS-11498. Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0. 2017-03-08 15:33:59 -08:00
Yiqun Lin b13878cf50 HDFS-8741. Proper error msg to be printed when invalid operation type is given to WebHDFS operations. Contributed by Surendra Singh Lilhore. 2017-03-08 15:33:58 -08:00
Mingliang Liu 370f4da168 HDFS-11494. Log message when DN is not selected for block replication. Contributed by Yiqun Lin 2017-03-08 15:33:56 -08:00
Jing Zhao 09cfa79dbd HDFS-11476. Fix NPE in FsDatasetImpl#checkAndUpdate. Contributed by Xiaobing Zhou. 2017-03-08 15:33:55 -08:00
Andrew Wang 4a09ed01a8 HDFS-8112. Relax permission checking for EC related operations. 2017-03-08 15:33:55 -08:00
Anu Engineer 4074123636 HDFS-11196. Ozone: Improve logging and error handling in the container layer. Contributed by Anu Engineer. 2017-03-08 09:36:20 -08:00
Andrew Wang 5addacb1e3 HDFS-11152. Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors. Contributed by SammiChen. 2017-03-08 08:47:38 -08:00
Anu Engineer dac2b9e7d9 HDFS-11509. Ozone: Fix TestEndpoint test regression. Contributed by Anu Engineer. 2017-03-07 22:38:11 -08:00
Anu Engineer f0f4c3e7e3 HDFS-11424. Block Storage: add container representation to meta data. Contributed Chen Liang. 2017-03-07 13:26:22 -08:00
Anu Engineer e37c5ed47d HDFS-11451. Ozone: Add protobuf definitions for container reports. Contributed by Anu Engineer. 2017-03-07 10:28:21 -08:00
Arpit Agarwal 959940b0ab HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-07 10:12:35 -08:00
Anu Engineer a13c9782a3 HDFS-11158. Ozone: TestDatanodeStateMachine is failing intermittently. Contributed by Weiwei Yang. 2017-03-07 10:02:16 -08:00
Anu Engineer 5d0a206a63 HDFS-11497. Ozone: Fix Container test regression. Contributed by Anu Engineer. 2017-03-06 21:20:30 -08:00
Arpit Agarwal b5adc5c301 HDFS-10838. Last full block report received time for each DN should be easily discoverable. Contributed by Surendra Singh Lilhore. 2017-03-06 16:39:53 -08:00
Andrew Wang 5e74196ede HDFS-11498. Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0. 2017-03-06 15:04:13 -08:00
Anu Engineer 72f0bed3dd HDFS-11487. Ozone: Exclude container protobuf files from findbugs check. Contributed by Yuanbo Liu. 2017-03-06 14:17:41 -08:00
Yiqun Lin 3536ce031c HDFS-8741. Proper error msg to be printed when invalid operation type is given to WebHDFS operations. Contributed by Surendra Singh Lilhore. 2017-03-06 19:04:03 +08:00
Mingliang Liu 0f336bab9c HDFS-11494. Log message when DN is not selected for block replication. Contributed by Yiqun Lin 2017-03-03 18:47:13 -08:00
Anu Engineer 0951726233 Merge branch 'trunk' into HDFS-7240 2017-03-03 14:03:35 -08:00
Jing Zhao ac5ae0065a HDFS-11476. Fix NPE in FsDatasetImpl#checkAndUpdate. Contributed by Xiaobing Zhou. 2017-03-03 13:31:20 -08:00
Andrew Wang 3085a60430 HDFS-8112. Relax permission checking for EC related operations. 2017-03-03 13:00:22 -08:00
Anu Engineer 3ab6bc5329 HDFS-11137. Ozone: SCM: Add negative tests cases for datanodeStatemachine. Contributed by Weiwei Yang. 2017-03-03 10:29:05 -08:00
Rakesh Radhakrishnan 3749152b66 HDFS-11416. Refactor out system default erasure coding policy. Contributed by Andrew Wang. 2017-03-03 09:28:03 +05:30
Andrew Wang 8f4817f2c5 Revert "HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru."
This reverts commit e61491d476.
2017-03-02 16:20:48 -08:00
Arpit Agarwal e61491d476 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-02 15:38:25 -08:00
Anu Engineer e79f9da8de HDFS-11484. Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate. Contributed by Xiaoyu Yao. 2017-03-02 15:10:47 -08:00
Arpit Agarwal b3ec531f40 HDFS-11461. DataNode Disk Outlier Detection. Contributed by Hanisha Koneru. 2017-03-02 12:45:48 -08:00
Anu Engineer 02c35065f7 HDFS-11480. Ozone: TestEndpoint task failure. Contributed by Xiaoyu Yao. 2017-03-02 11:18:41 -08:00
Anu Engineer 09ad229db9 HDFS-11425. Ozone: add client-facing container APIs and container references. Contributed by Chen Liang. 2017-03-02 11:00:25 -08:00
Arpit Agarwal eeca8b0c4e HDFS-11419. HDFS specific network topology classes with storage type info included. Contributed by Chen Liang. 2017-03-02 09:21:27 -08:00
Ming Ma 4e14eaded2 Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/hadoop into trunk 2017-03-01 20:24:49 -08:00
Ming Ma 25c84d279b HDFS-11412. Maintenance minimum replication config value allowable range should be [0, DefaultReplication]. (Manoj Govindassamy via mingma) 2017-03-01 20:23:52 -08:00
Yiqun Lin 555d0c3995 HDFS-11478. Update EC commands in HDFSCommands.md. Contributed by Yiqun Lin. 2017-03-02 12:10:24 +08:00
Anu Engineer 7aa0a44cf1 HDFS-11447. Ozone: SCM: Send node report to SCM with heartbeat.Contributed by Xiaoyu Yao. 2017-03-01 11:47:22 -08:00
Anu Engineer 3eae84c1be HDFS-11474. Ozone: TestContainerMapping needs to cleanup levelDB files.Contributed by Xiaoyu Yao. 2017-03-01 11:30:32 -08:00
Anu Engineer 4cd34e2a18 HDFS-11473. Ozone : exclude cblock protobuf files from findbugs check. Contributed by Xiaoyu Yao. 2017-03-01 09:40:18 -08:00
Rakesh Radhakrishnan 82ef9accaf HDFS-11428. Change setErasureCodingPolicy to take a required string EC policy name. Contributed by Andrew Wang. 2017-03-01 16:06:02 +05:30
Brahma Reddy Battula dcd03df9f9 HDFS-11432. Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands. Contributed by Brahma Reddy Battula. 2017-03-01 10:45:56 +05:30
Anu Engineer 00684d62ca HDFS-11036. Ozone: reuse Xceiver connection. Contributed by Chen Liang. 2017-02-28 12:15:26 -08:00
Arpit Agarwal ad16978e65 HDFS-11414. Ozone : move StorageContainerLocation protocol to hdfs-client. Contributed by Chen Liang. 2017-02-28 10:28:05 -08:00
Zhe Zhang d269b488a7 HDFS-11466. Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms. Contributed by Andrew Wang. 2017-02-28 09:00:49 -08:00
Akira Ajisaka 258342e76c HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR. 2017-02-28 13:47:44 +09:00
Andrew Wang 55c07bbed2 HDFS-11382. Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage. Contributed by Manoj Govindassamy. 2017-02-27 17:07:45 -08:00
Anu Engineer d63ec0ca81 HDFS-11184. Ozone: SCM: Make SCM use container protocol. Contributed by Anu Engineer. 2017-02-27 12:25:03 -08:00
Anu Engineer ae783b1991 HDFS-11463. Ozone: Add metrics for container operations and export over JMX. Contributed by Mukul Kumar Singh. 2017-02-27 09:44:48 -08:00
Wei-Chiu Chuang 05391c1845 HDFS-10506. OIV's ReverseXML processor cannot reconstruct some snapshot details. Contributed by Akira Ajisaka. 2017-02-25 14:38:50 -08:00
Arpit Agarwal 120bef7de8 HDFS-11462. Fix occasional BindException in TestNameNodeMetricsLogger. 2017-02-24 19:53:39 -08:00
Yiqun Lin 4a58870a04 HDFS-11433. Document missing usages of OfflineEditsViewer processors. Contributed by Yiqun Lin. 2017-02-25 10:00:05 +08:00
Xiao Chen e24ed47d9a HDFS-11421. Make WebHDFS' ACLs RegEx configurable. Contributed by Harsh J. 2017-02-24 16:49:46 -08:00
Arpit Agarwal d2b3ba9b8f HDFS-11295. Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete(). Contributed by Marton Elek. 2017-02-24 15:44:11 -08:00
Andrew Wang c1a52b04d0 HDFS-11427. Rename rs-default to rs. 2017-02-24 10:58:45 -08:00
Andrew Wang 132f758e3d HDFS-11426. Refactor EC CLI to be similar to storage policies CLI. 2017-02-23 16:00:11 -08:00
Jing Zhao 13d4bcfe35 HDFS-4025. QJM: Sychronize past log segments to JNs that missed them. Contributed by Hanisha Koneru. 2017-02-22 16:33:38 -08:00
Andrew Wang d150f061f4 HDFS-11438. Fix typo in error message of StoragePolicyAdmin tool. Contributed by Alison Yu. 2017-02-22 15:16:09 -08:00
Ming Ma cfcd527323 HDFS-11411. Avoid OutOfMemoryError in TestMaintenanceState test runs. (Manoj Govindassamy via mingma) 2017-02-22 09:41:07 -08:00
Mingliang Liu 003ae00693 HDFS-11430. Separate class InnerNode from class NetworkTopology and make it extendable. Contributed by Tsz Wo Nicholas Sze 2017-02-21 15:32:46 -08:00
Andrew Wang 8ef7ebbc71 HDFS-11406. Remove unused getStartInstance and getFinalizeInstance in FSEditLogOp. Contributed by Alison Yu. 2017-02-21 14:54:20 -08:00
Andrew Wang fc9ad3ce3a HDFS-11405. Rename "erasurecode" CLI subcommand to "ec". Contributed by Manoj Govindassamy. 2017-02-21 13:55:27 -08:00
Eric Payne a07ddef101 HDFS-11404. Increase timeout on TestShortCircuitLocalRead.testDeprecatedGetBlockLocalPathInfoRpc. Contributed by Eric Badger 2017-02-21 12:04:25 -06:00
Vinayakumar B 4804050630 HDFS-11177. 'storagepolicies -getStoragePolicy' command should accept URI based path. (Contributed by Surendra Singh Lilhore) 2017-02-21 18:13:19 +05:30
Xiao Chen 02c549484a HDFS-11410. Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp. 2017-02-16 18:07:55 -08:00
Wei-Chiu Chuang 5690b51ef7 HDFS-11100. Recursively deleting file protected by sticky bit should fail. Contributed by John Zhuge. 2017-02-16 05:39:37 -08:00
Ming Ma a136936d01 HDFS-11265. Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI. (Marton Elek via mingma) 2017-02-15 20:24:07 -08:00
Mingliang Liu 0741dd3b9a HDFS-11375. Display the volume storage type in datanode UI. Contributed by Surendra Singh Lilhore 2017-02-15 11:37:26 -08:00
Jing Zhao 627da6f717 HDFS-8498. Blocks can be committed with wrong size. Contributed by Jing Zhao. 2017-02-15 10:44:37 -08:00