Commit Graph

7598 Commits

Author SHA1 Message Date
Chen Liang caeeb78ca4 HDFS-12799. Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 966853894f HDFS-12795. Ozone: SCM: Support for Container LifeCycleState PENDING_CLOSE and LifeCycleEvent FULL_CONTAINER. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 26e270b908 HDFS-12751. Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer fd09c2ce5b HDFS-12890 . Ozone: XceiverClient should have upper bound on async requests. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer 2f3068bc72 HDFS-12698. Ozone: Use time units in the Ozone configuration values. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 8ff98e2af3 HDFS-12000. Ozone: Container : Add key versioning support-1. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao c0c87dea9b HDFS-12626. Ozone : delete open key entries that will no longer be closed. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer 6c630d0b09 HDFS-12899. Ozone: SCM: BlockManagerImpl close is called twice during StorageContainerManager#stop. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 74362cb503 HDFS-12901. Ozone: SCM: Expose StorageContainerManager#getScmId through container protocol. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar 11473b8204 HDFS-12898. Ozone: TestSCMCli#testHelp and TestSCMCli#testNonExistCommand fail consistently. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer 207523915d HDFS-12588. Use GenericOptionsParser for scm and ksm daemon. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Nanda kumar 7bba3943f0 HDFS-12876. Ozone: moving NodeType from OzoneConsts to Ozone.proto. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 85ec48a646 HDFS-12867. Ozone: TestOzoneConfigurationFields fails consistently. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer 538e17824d HDFS-12850. Ozone: TestContainerPersistence#testListContainer fails frequently due to timed out. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer e32e187448 HDFS-12864. Ozone: Turn off full synced write in RocksDB MetadataStore. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 8023ba50b5 HDFS-12807. Ozone: Expose RockDB stats via JMX for Ozone metadata stores. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze 86802a461c HDFS-12851. Ozone: Upgrade to latest ratis build. Contributed by Mukul Kumar Singh 2018-04-26 05:36:04 -07:00
Nanda kumar ea86945754 HDFS-12842. Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 73ca8608da HDFS-12786. Ozone: add port/service names to the ksm/scm web ui. Contributed by Elek Marton. 2018-04-26 05:36:04 -07:00
Nanda kumar f9c11d952c HDFS-12740. SCM should support a RPC to share the cluster Id with KSM and DataNodes. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao c8d8270f72 HDFS-12787. Ozone: SCM: Aggregate the metrics from all the container reports. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Yiqun Lin cbe4f314b3 HDFS-12731. Ozone: Fix inconsistent default value in ozone settings. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Anu Engineer 4deb845d7c HDFS-12759. Ozone: web: integrate configuration reader page to the SCM/KSM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 8e8ff85b16 HDFS-12830. Ozone: TestOzoneRpcClient#testPutKeyRatisThreeNodes fails. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Yiqun Lin 214b52dfce HDFS-12500. Ozone: add logger for oz shell commands and move error stack traces to DEBUG level. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 419e86d580 HDFS-12805. Ozone: Redundant characters printed in exception log. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer 93f1b04949 HDFS-12798. Ozone: scm web: fix the node status table. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 4460ac9c57 HDFS-12756. Ozone: Add datanodeID to heartbeat responses and container protocol. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Nanda kumar 446e84357d HDFS-12796. SCM should not start if Cluster Version file does not exist. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer 740a06cdd7 HDFS-12719. Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Yiqun Lin 761926db2c HDFS-12793. Ozone : TestSCMCli is failing consistently. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Chen Liang e395244e95 HDFS-12758. Ozone: Correcting assertEquals argument order in test cases. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao cec96b296f HDFS-12549. Ozone: OzoneClient: Support for REST protocol. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar 9734f505ea HDFS-12739. Add Support for SCM --init command. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 709d56fdc9 HDFS-12474. Ozone: SCM: Handling container report with key count and container usage. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 2ec24a7542 HDFS-12774. Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Yiqun Lin 132f30c8ea HDFS-12443. Ozone: Improve SCM block deletion throttling algorithm. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh 7ebe79e879 HDFS-12720. Ozone: Ratis options are not passed from KSM Client protobuf helper correctly. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Weiwei Yang 0559265822 HDFS-12750. Ozone: Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 0760418bec HDFS-12697. Ozone services must stay disabled in secure setup for alpha. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh b687902590 HDFS-12718. Ozone: fix thread number calculation in CBlockManager. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 76d34bca62 HDFS-12521. Ozone: SCM should read all Container info into memory when booting up. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Anu Engineer 6291ca1e32 HDFS-12663. Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Nanda kumar 80357c0eee HDFS-12678. Ozone: Corona: Add statistical information to json output. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Anu Engineer 1b56a73a4f HDFS-12680. Ozone: SCM: Lease support for container creation. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 338c909ac6 HDFS-12558. Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 333dae888c HDFS-12690. Ozone: generate swagger descriptor for the Ozone REST Api. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 0582d8b1b1 Revert "HDFS-12558. Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui. Contributed by Elek, Marton."
This reverts commit 18c0cdcec9.
2018-04-26 05:36:04 -07:00
Anu Engineer 402a1c1ba3 HDFS-12558. Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 731a69faa1 HDFS-12693. Ozone: Enable XFrame options for KSM/SCM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Yiqun Lin 6783dad766 HDFS-11468. Ozone: SCM: Add Node Metrics for SCM. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Yiqun Lin 393a02d8e1 HDFS-12692. Ozone: fix javadoc/unit test for calling MetadataStore.getRangeKVs with non existing key. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Weiwei Yang 6d9f914b55 HDFS-12684. Ozone: SCMMXBean NodeCount is overlapping with NodeManagerMXBean. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Nanda kumar cc86719451 HDFS-12675. Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Weiwei Yang 0f0417a12c HDFS-12673. Ozone: Log elapsed time for block deletion tasks. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer 06c6582694 HDFS-12585. Ozone: KSM UI: Add description for configs in UI. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 763484e596 HDFS-12661. Ozone: Support optional documentation link in KSM/SCM webui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao a467017068 HDFS-12411. Ozone: Add container usage information to DN container report. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh fb545e4291 HDFS-12415. Ozone: TestXceiverClientManager and TestAllocateContainer occasionally fails. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Chen Liang 895a4cc60a HDFS-12632. Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Nanda kumar 9bb516118d HDFS-12490. Ozone: OzoneClient: Add creation/modification time information in OzoneVolume/OzoneBucket/OzoneKey. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Nandakumar 8881309002 HDFS-12519. Ozone: Lease Manager framework. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer ebc4d4ebc3 HDFS-12475. Ozone: add documentation about Datanode http address. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh c85d3b2d25 HDFS-12593. Ozone: update Ratis to the latest snapshot. Contributed by Tsz Wo Nicholas Sze. 2018-04-26 05:36:04 -07:00
Nandakumar 5ed663b32f HDFS-12631. Ozone: ContainerStorageLocation#scmUsage should count only SCM usage. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Yiqun Lin d19b4c8763 HDFS-12583. Ozone: Fix swallow exceptions which makes hard to debug failures. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 9445a9267f HDFS-12572. Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Chen Liang ae2b77a0e5 HDFS-12546. Ozone: DB listing operation performance improvement. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Chen Liang b05ad0b315 HDFS-12611. Ozone: SCM Cli: Use the hostname of SCM server instead of the bind address. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
yuanbo e76e10fd7a HDFS-12609. Ozone: SCM: Refactoring of chill mode logic in NodeManager. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze e3b51d9074 HDFS-12598. Ozone: Fix 3 node ratis replication in Ozone. Contributed by Mukul Kumar Singh 2018-04-26 05:36:04 -07:00
Nandakumar 3504af9411 HDFS-12537. Ozone: Reduce key creation overhead in Corona. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Nandakumar 9630621be7 HDFS-12610. Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Nandakumar c3ef381011 HDFS-12543. Ozone : allow create key without specifying size. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer c97ce55d06 HDFS-12602. Add Ozone config UI to KSM and SCM index page. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 303ecae5d5 HDFS-12557. Ozone: Improve the formatting of the RPC stats on web UI. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer c17521b1bd HDFS-12387. Ozone: Support Ratis as a first class replication mechanism. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Anu Engineer 0b3270a8de HDFS-12513. Ozone: Create UI page to show Ozone configs by tag. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer c52019c291 HDFS-12539. Ozone: refactor some functions in KSMMetadataManagerImpl to be more readable and reusable. Contributed by Yuanbo Liu. 2018-04-26 05:36:04 -07:00
Anu Engineer ceec14bff5 HDFS-12467. Ozone: SCM: NodeManager should log when it comes out of chill mode. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer d7a94a21ae HDFS-12471. Ozone: Reduce some KSM/SCM deletion log messages from INFO to DEBUG. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Anu Engineer 6a754caca0 HDFS-12568. Ozone: Cleanup the ozone-default.xml. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 05246e2b32 HDFS-12060. Ozone: OzoneClient: Add list calls in RpcClient. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 7846636a5d HDFS-12551. Ozone: Documentation: Add Ozone overview documentation. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Yiqun Lin cb1fb3436e HDFS-12563. Ozone: Creation time wrongly showed in OzoneRest.md. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer aaec2689f4 HDFS-12511. Ozone: Add tags to config. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 7151a9b604 HDFS-12488. Ozone: OzoneRestClient timeout is not configurable. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Weiwei Yang 7363c8fc09 HDFS-12540. Ozone: node status text reported by SCM is a bit confusing. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 13fdb58490 HDFS-12554. Ozone: Fix TestDatanodeStateMachine#testDatanodeStateMachineWithInvalidConfiguration. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Chen Liang 9530153f33 HDFS-12268. Ozone: Add metrics for pending storage container requests. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer 94090acf3a HDFS-12469. Ozone: Create docker-compose definition to easily test real clusters. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer a44f3a777d HDFS-12501. Ozone: Cleanup javac issues. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Yiqun Lin 6b10723ec3 HDFS-12525. Ozone: OzoneClient: Verify bucket/volume name in create calls. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 5e4a6b686c HDFS-12425. Ozone: OzoneFileSystem: read/write/create/open/getFileInfo APIs. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer a1a3ba6529 HDFS-12454. Ozone : the sample ozone-site.xml in OzoneGettingStarted does not work. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer 2804435386 HDFS-12017. Ozone: Container: Move IPC port to 98xx range. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao b2447f8dd9 HDFS-11614. Ozone: Cleanup javadoc issues. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Chen Liang 9ff136bb02 HDFS-12210. Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Chen Liang dddded0016 Revert "Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh." to fix commit message.
This reverts commit 087c69ba24.
2018-04-26 05:36:04 -07:00
Weiwei Yang d303b7f40f HDFS-12524. Ozone: Record number of keys scanned and hinted for getRangeKVs call. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00