Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,923 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

785 Commits

Author SHA1 Message Date
Xiaoyu Yao 0e450202a6
HDDS-1019. Use apache/hadoop-runner image to test ozone secure cluster. Contributed by Xiaoyu Yao. 2019-02-27 12:44:04 +01:00
Ajay Kumar 625e93713b HDDS-1177. Add validation to AuthorizationHeaderV4. Contributed by Ajay Kumar 2019-02-26 18:57:43 -08:00
Anu Engineer a5a751b418 HDDS-1174. Freon tests are failing with null pointer exception. 
Contributed by Shashikant Banerjee.
 2019-02-26 14:33:54 -08:00
Márton Elek 585cebf26b
HDDS-1104. Use picocli with Ozone genesis tool. Contributed by Lokesh Jain. 2019-02-26 16:11:55 +01:00
Xiaoyu Yao 9e0f3d1c52 HDDS-1038. Support Service Level Authorization for Ozone. Contributed by Xiaoyu Yao and Ajay Kumar. 2019-02-26 06:02:10 -08:00
Anu Engineer 9de34d2990 HDDS-1151. Propagate the tracing id in ScmBlockLocationProtocol. 
Contributed by Elek, Marton.
 2019-02-25 13:50:37 -08:00
Márton Elek a5eb7ce651
HDDS-1170. Fix the FindBugs issue in OzoneClientAdapterFactory. Contributed by Anu Engineer. 2019-02-25 10:41:43 +01:00
Anu Engineer 92b1fdcece HDDS-1070. Adding Node and Pipeline related metrics in SCM. 
Contributed by Nandakumar.
 2019-02-24 15:39:41 -08:00
Anu Engineer 3aa0a57ea0 HDDS-1152. Add trace information for the client side of the datanode writes 
Contributed by Elek, Marton.
 2019-02-24 15:20:34 -08:00
Anu Engineer 5c1f946071 HDDS-1145. Add optional web server to the Ozone freon test tool. 
Contributed by Elek, Marton.
 2019-02-24 15:04:25 -08:00
Anu Engineer 236b105e41 HDDS-1089. Disable OzoneFSStorageStatistics for hadoop versions older than 2.8. 
Contributed by Elek, Marton.
 2019-02-24 13:53:38 -08:00
Anu Engineer 8387bbdfb5 HDDS-1154. Add tracing to the client side of StorageContainerLocationProtocol and OzoneManagerProtocol. 
Contributed by Elek, Marton.
 2019-02-24 13:13:37 -08:00
Bharat Viswanadham 106bdc6c04
HDDS-1120. Add a config to disable checksum verification during read. 2019-02-23 21:37:55 -08:00
Anu Engineer c1e5b19212 HDDS-1147. Fix failing unit tests in TestOzoneManager. 
Contributed by Nandakumar.
 2019-02-23 20:56:24 -08:00
Anu Engineer fb546d1807 HDDS-1167. Error in hadoop-ozone/dev-support/checks/checkstyle.sh. 
Contributed by Nandakumar.
 2019-02-23 20:45:01 -08:00
Hanisha Koneru d0e21ac16b HDDS-1158. TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error. 2019-02-23 10:20:28 -08:00
Elek, Márton 28d0bf9f67 HDDS-1161. Disable failing test which are tracked by a separated jira. Contributed by Elek, Marton. 2019-02-21 13:57:05 -08:00
Bharat Viswanadham d33f0666f6
HDDS-1141. Update DBCheckpointSnapshot to DBCheckpoint. 
* HDDS-1141.Update DBCheckpointSnapshot to DBCheckpoint.

* fix test failures in TestOzoneConfigurationFields
 2019-02-21 13:29:10 -08:00
Márton Elek 7c802c42dc
HDDS-1129. Fix findbug/checkstyle errors hdds projects. Contributed by Elek, Marton. 2019-02-21 14:04:18 +01:00
Hanisha Koneru b1397ff9e4 HDDS-1109. Setup Failover Proxy Provider for OM client. 2019-02-20 14:49:59 -08:00
Arpit Agarwal 676a9cb888 HDDS-1053. Generate RaftGroupId from OMServiceID. Contributed by Aravindan Vijayan. 2019-02-20 12:57:49 -08:00
Xiaoyu Yao 1374f8f548 HDDS-1060. Add API to get OM certificate from SCM CA. Contributed by Ajay Kumar. 2019-02-20 11:35:49 -08:00
Márton Elek 642fe6a260
HDDS-1135. Ozone jars are missing in the Ozone Snapshot tar. Contributed by Dinesh Chitlangia. 2019-02-20 12:08:24 +01:00
Yiqun Lin 1d30fd94c6 HDDS-1130. Make BenchMarkBlockManager multi-threaded. Contributed by Lokesh Jain. 2019-02-20 10:45:51 +08:00
avijayanhwx e8d7e3b4e6 HDDS-1139 : Fix findbugs issues caused by HDDS-1085. Contributed by Aravindan Vijayan. 2019-02-19 16:32:47 -08:00
Anu Engineer 02d04bd107 HDDS-1121. Key read failure when data is written parallel in to Ozone. 
Contributed by Bharat Viswanadham.
 2019-02-19 11:07:17 -08:00
Xiaoyu Yao cf1a66d0da HDDS-1101. SCM CA: Write Certificate information to SCM Metadata. Contributed by Anu Engineer. 2019-02-19 10:42:56 -08:00
Anu Engineer 588b4c4d78 HDDS-1085. Create an OM API to serve snapshots to Recon server. 
Contributed by Aravindan Vijayan.
 2019-02-18 22:35:36 -08:00
Yiqun Lin 67af509097 HDDS-1122. Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure. Contributed by Yiqun Lin. 2019-02-19 11:29:52 +08:00
Yiqun Lin f2fb6536dc HDDS-1106. Introduce queryMap in PipelineManager. Contributed by Lokesh Jain. 2019-02-18 22:35:23 +08:00
Anu Engineer 7ea91494ce HDDS-1041. Support TDE(Transparent Data Encryption) for Ozone. 
Contributed by Xiaoyu Yao.
 2019-02-16 10:23:10 -08:00
Elek, Márton afe126d71f HDDS-1114. Fix findbugs/checkstyle/accepteance errors in Ozone. Contributed by Marton Elek. 2019-02-15 11:49:52 -08:00
Anu Engineer 217bdbd940 HDDS-1116.Add java profiler servlet to the Ozone web servers. 
Contributed by Elek, Marton.
 2019-02-15 11:32:06 -08:00
Elek, Márton d10444e79c
HDDS-1092. Use Java 11 JRE to run Ozone in containers. 2019-02-15 17:26:56 +01:00
Shashikant Banerjee de934ba2dc HDDS-1076. TestSCMNodeManager crashed the jvm. Contributed by Lokesh Jain. 2019-02-15 21:13:30 +05:30
Márton Elek e0fe3d1eca HDDS-1110. OzoneManager need to login during init when security is enabled. Contributed by Xiaoyu Yao. 2019-02-15 16:12:24 +01:00
Anu Engineer 75e15cc0c4 HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects. 
Contributed by Elek, Marton.
 2019-02-14 23:33:25 -08:00
Anu Engineer 5cb67cf044 HDDS-1097. Add genesis benchmark for BlockManager#allocateBlock. 
Contributed by Lokesh Jain.
 2019-02-14 23:24:58 -08:00
Anu Engineer 506bd02c63 HDDS-905. Create informative landing page for Ozone S3 gateway. 
Contributed by Elek, Marton.
 2019-02-14 23:16:23 -08:00
Anu Engineer 0395f22145 HDDS-1068. Improve the error propagation for ozone sh. 
Contributed by Elek, Marton.
 2019-02-14 22:51:46 -08:00
Yiqun Lin 084b6a6751 HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham. 2019-02-15 14:23:34 +08:00
Yiqun Lin 492e49e7ca Revert "HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham." 
This reverts commit 5656409327.
 2019-02-15 14:21:29 +08:00
Yiqun Lin 5656409327 HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham. 2019-02-15 14:07:15 +08:00
Anu Engineer 2d83b24994 HDDS-1108. Check s3bucket exists or not before MPU operations. 
Contributed by Bharat Viswanadham.
 2019-02-14 18:10:18 -08:00
Anu Engineer 6c8ffdb958 HDDS-1100. fix asf license errors in newly added files by HDDS-936. 
Contributed by  Dinesh Chitlangia.
 2019-02-14 17:42:59 -08:00
Márton Elek 7a57974f1f HDDS-1096. OzoneManager#loadOMHAConfigs should use default ports in case port is not defined. Contributed by Hanisha Koneru. 2019-02-14 14:02:16 +01:00
Jitendra Pandey fa067aa157 HDDS-936. Need a tool to map containers to ozone objects. Contributed by Sarun Singla 2019-02-13 15:43:14 -08:00
Bharat Viswanadham cf4aeccfa0 HDDS-1034. TestOzoneRpcClient and TestOzoneRpcClientWithRatis failure. Contributed by Mukul Kumar Singh. 2019-02-12 22:23:48 -08:00
Bharat Viswanadham 917ac9f108 HDDS-972. Add support for configuring multiple OMs. Contributed by Hanisha Koneru. 2019-02-12 22:07:27 -08:00
Yiqun Lin 06d7890bdd HDDS-1047. Fix TestRatisPipelineProvider#testCreatePipelineWithFactor. Contributed by Nilotpal Nandi. 2019-02-13 10:50:57 +08:00
Mukul Kumar Singh 20b92cdefe HDDS-1050. TestSCMRestart#testPipelineWithScmRestart is failing. Contributed by Supratim Deka. 2019-02-12 16:33:58 +05:30
Nanda kumar 63a9b0d8aa HDDS-1080. Ozonefs Isolated class loader should support FsStorageStatistics. 
Contributed by Elek, Marton.
 2019-02-12 14:49:51 +05:30
Nanda kumar a536eb5c41 HDDS-360. Use RocksDBStore and TableStore for SCM Metadata. 
Contributed by Anu Engineer.
 2019-02-12 14:26:19 +05:30
Yiqun Lin 26e60135f5 HDDS-1074. Remove dead variable from KeyOutputStream#addKeyLocationInfo. Contributed by Siddharth Wagle. 2019-02-12 16:28:50 +08:00
Ajay Kumar d48e61dd36 HDDS-1012. Add Default CertificateClient implementation. Contributed by Ajay Kumar 2019-02-11 18:59:47 -08:00
Mukul Kumar Singh 73b67b2df5 HDDS-1040. Add blockade Tests for client failures. Contributed by Nilotpal Nandi. 2019-02-11 20:08:25 +05:30
Márton Elek e7d1ae52d2 HDDS-1017. Use distributed tracing to indentify performance problems in Ozone. Contributed by Elek, Marton. 2019-02-11 12:18:26 +01:00
Bharat Viswanadham a1414582a8 HDDS-1077. TestSecureOzoneCluster does not config OM HTTP keytab. Contributed by Xiaoyu Yao. 2019-02-10 19:34:53 -08:00
Bharat Viswanadham ed99da850e HDDS-1078. TestRatisPipelineProvider failing because of node count mismatch. Contributed by Mukul Kumar Singh. 2019-02-10 19:17:08 -08:00
Shashikant Banerjee 965d26c9c7 HDDS-1026. Reads should fail over to alternate replica. Contributed by Shashikant Banerjee. 2019-02-10 10:53:16 +05:30
Xiaoyu Yao e50dc7ee59 HDDS-1018. Update the base image of krb5 container for the secure ozone cluster. Contributed by Xiaoyu Yao. 2019-02-08 12:45:49 -08:00
Nanda kumar 394a9f7d5c HDDS-1033. Add FSStatistics for OzoneFileSystem. 
Contributed by Mukul Kumar Singh.
 2019-02-08 17:33:37 +05:30
Márton Elek df7b7dadf9 HDDS-1073. Fix FindBugs issues on OzoneBucketStub#createMultipartKey. Contributed by Aravindan Vijayan. 2019-02-08 11:35:55 +01:00
Nanda kumar 0c1bc4dcee HDDS-981. Block allocation should involve pipeline selection and then container selection. 
Contributed by Lokesh Jain.
 2019-02-08 15:44:41 +05:30
Bharat Viswanadham a140a890c6 HDDS-1069. Temporarily disable the security acceptance tests by default in Ozone. Contributed by Marton Elek. 2019-02-07 18:05:05 -08:00
Bharat Viswanadham 75e8441c61 HDDS-1071. Make Ozone s3 acceptance test suite centos compatible. Contributed by Elek Marton. 2019-02-07 08:53:11 -08:00
Márton Elek a65aca2fef HDDS-922. Create isolated classloder to use ozonefs with any older hadoop versions. Contributed by Elek, Marton. 2019-02-07 17:02:03 +01:00
Mukul Kumar Singh 214112b2d7 HDDS-1010. ContainerSet#getContainerMap should be renamed. Contributed by Supratim Deka. 2019-02-07 18:06:23 +05:30
Mukul Kumar Singh 911790cc26 HDDS-1027. Add blockade Tests for datanode isolation and scm failures. Contributed by Nilotpal Nandi. 2019-02-06 11:32:38 +05:30
Márton Elek 194f0b49fb HDDS-631. Ozone classpath shell command is not working. Contributed by Elek, Marton. 2019-02-05 16:20:30 +01:00
Márton Elek 9f2da01591 HDDS-776. Make OM initialization resilient to dns failures. Contributed by Doroszlai, Attila. 2019-02-05 16:11:19 +01:00
Márton Elek 9ace37b119 HDDS-987. MultipartUpload: S3API for list parts of a object. Contributed by Bharat Viswanadham. 2019-02-05 13:08:36 +01:00
Yiqun Lin 3efa168e1f HDDS-1029. Allow option for force in DeleteContainerCommand. Contributed by Bharat Viswanadham. 2019-02-05 10:51:52 +08:00
Mukul Kumar Singh b6f90d3957 HDDS-1021. Allow ozone datanode to participate in a 1 node as well as 3 node Ratis Pipeline. Contributed by Shashikant Banerjee. 2019-02-03 00:53:06 +05:30
Shashikant Banerjee 4123353151 HDDS-1037. Fix the block discard logic in Ozone client. Contributed by Shashikant Banerjee. 2019-02-01 20:44:48 +05:30
Mukul Kumar Singh 13aa939224 HDDS-997. Add blockade Tests for scm isolation and mixed node isolation. Contributed by Nilotpal Nandi. 2019-02-01 13:10:17 +05:30
Márton Elek 033d97abac HDDS-956. MultipartUpload: List Parts for a Multipart upload key. Contributed by Bharat Viswanadham. 2019-01-31 14:51:31 +01:00
Bharat Viswanadham 53729279c7 HDDS-1035. Intermittent TestRootList failure. Contributed by Doroszlai Attila. 2019-01-30 16:21:42 -08:00
Bharat Viswanadham 945a61c164 HDDS-549. Add support for key rename in Ozone Shell. Contributed by Doroszlai Attila. 2019-01-30 16:00:18 -08:00
Bharat Viswanadham 7456fc99ee HDDS-1031. Update ratis version to fix a DN restart Bug. Contributed by Bharat Viswanadham. 2019-01-30 11:14:02 -08:00
Márton Elek 0e95ae402c HDDS-1030. Move auditparser robot tests under ozone basic. Contributed by Dinesh Chitlangia. 2019-01-30 17:34:32 +01:00
Yiqun Lin d583cc45c6 HDDS-1024. Handle DeleteContainerCommand in the SCMDatanodeProtocolServer. Contributed by Bharat Viswanadham. 2019-01-30 13:56:28 +08:00
Yiqun Lin 8326450bca HDDS-974. Add getServiceAddress method to ServiceInfo and use it in TestOzoneShell. Contributed by Doroszlai, Attila. 2019-01-28 17:03:12 +08:00
Hanisha Koneru 91649c34aa HDDS-1013. NPE while listing directories. 2019-01-25 21:39:03 -08:00
Bharat Viswanadham fb69519e68 HDDS-1011. Fix NPE BucketManagerImpl.setBucketProperty. Contributed by Xiaoyu Yao. 2019-01-25 14:46:17 -08:00
Xiaoyu Yao 2ec296e659 HDDS-991. Fix failures in TestSecureOzoneCluster. Contributed by Ajay Kumar. 2019-01-25 10:50:23 -08:00
Xiaoyu Yao 9fc7df8afb HDDS-793. Support custom key/value annotations on volume/bucket/key. Contributed by Elek, Marton. 2019-01-24 23:24:09 -08:00
Nanda kumar a448b05287 HDDS-1009. TestAbortMultipartUpload is missing the apache license text. 
Contributed by Dinesh Chitlangia.
 2019-01-25 12:20:12 +05:30
Nanda kumar 45c4cfe790 HDDS-906. Display the ozone version on SCM/OM web ui instead of Hadoop version. 
Contributed by Doroszlai, Attila.
 2019-01-25 12:16:29 +05:30
Nanda kumar 8ff9578126 HDDS-1007. Add robot test for AuditParser. 
Contributed by Dinesh Chitlangia.
 2019-01-25 12:14:10 +05:30
Nanda kumar c6d901af77 HDDS-1006. AuditParser assumes incorrect log format. 
Contributed by Dinesh Chitlangia.
 2019-01-25 12:11:58 +05:30
Márton Elek 4e0aa2ceac HDDS-948. MultipartUpload: S3 API for Abort Multipart Upload. Contributed by Bharat Viswanadham. 2019-01-24 20:24:19 +01:00
Márton Elek 3c7d700b65 HDDS-1002. ozonesecure compose incompatible with smoke test. Contributed by Doroszlai, Attila. 2019-01-24 17:17:40 +01:00
Shashikant Banerjee f3d8265582 HDDS-996. Incorrect data length gets updated in OM by client in case it hits exception in multiple successive block writes. Contributed by Shashikant Banerjee. 2019-01-24 16:37:05 +05:30
Márton Elek 09a5859f58 HDDS-993. Update hadoop version to 3.2.0. Contributed by Supratim Deka. 2019-01-24 11:32:58 +01:00
Ajay Kumar dcbc8b86ed HDDS-975. Manage ozone security tokens with ozone shell cli. Contributed by Ajay Kumar. 2019-01-23 15:34:20 -08:00
Elek, Márton 0b91329ed6 HDDS-764. Run S3 smoke tests with replication STANDARD. (#462) 
HDDS-764. Run S3 smoke tests with replication STANDARD. Contributed by Elek, Marton.
 2019-01-23 11:37:49 -08:00
Shashikant Banerjee e72e27edd8 HDDS-932. Add blockade Tests for Network partition. Contributed by Nilotpal Nandi. 2019-01-23 14:05:49 +05:30
Xiaoyu Yao 751bc62df7 Merge branch 'HDDS-4' into trunk 2019-01-18 14:44:24 -08:00
Bharat Viswanadham c26d354e7d HDDS-983. Rename S3Utils to avoid conflict with HDFS classes. Contributed by Bharat Viswanadham. 2019-01-18 13:16:38 -08:00
Xiaoyu Yao 2aaaf12f9e HDDS-943. Add block token validation in HddsDispatcher/XceiverServer. Contributed by Ajay Kumar. 2019-01-18 12:11:34 -08:00
Mukul Kumar Singh 4ac0404fe0 HDDS-959. KeyOutputStream should handle retry failures. Contributed by Lokesh Jain. 2019-01-18 11:08:43 -08:00
Xiaoyu Yao 06c83d3b8a HDDS-984. Fix TestOzoneManagerRatisServer.testIsReadOnlyCapturesAllCmdTypeEnums. Contributed by Xiaoyu Yao. 2019-01-17 14:03:46 -08:00
Bharat Viswanadham c0683ed253 HDDS-960. Add cli command option for getS3Secret. Contributed by Dinesh Chitlangia. 2019-01-17 10:30:45 -08:00
Ajay Kumar 01a7f9ed35 HDDS-597. Ratis: Support secure gRPC endpoint with mTLS for Ratis. Contributed by Ajay Kumar. 2019-01-17 10:30:44 -08:00
Bharat Viswanadham 54b11de2c0 HDDS-898. Continue token should contain the previous dir in Ozone s3g object list. Contributed by Elek Marton. 2019-01-16 17:33:10 -08:00
Xiaoyu Yao 140565f4db HDDS-967. Fix failures in TestOzoneConfigurationFields. Contributed by Ajay Kumar. 2019-01-16 16:44:33 -07:00
Ajay Kumar 0e16cf1973 HDDS-970. Fix classnotfound error for bouncy castle classes in OM,SCM init. Contributed by Ajay Kumar. 2019-01-16 16:40:05 -07:00
Ajay Kumar 0faa5701d9 HDDS-964. Fix test failure in TestOmMetrics. Contributed by Ajay Kumar. 2019-01-16 16:40:04 -07:00
Anu Engineer 8978466fca HDDS-945. Fix generics warnings in delegation token. 
Contributed by Ajay Kumar.
 2019-01-16 16:23:26 -07:00
Anu Engineer 1d5734e341 HDDS-938. Add Client APIs for using S3 Auth interface. 
Contributed by  Dinesh Chitlangia.
 2019-01-16 16:18:39 -07:00
Anu Engineer 30bfc9cbd0 HDDS-955. SCM CA: Add CA to SCM. 
Contributed by Anu Engineer.
 2019-01-16 13:43:31 -07:00
Bharat Viswanadham ddaef67181 HDDS-937. Create an S3 Auth Table. Contributed by Dinesh Chitlangia. 2019-01-16 13:43:30 -07:00
Ajay Kumar 50c4045fde HDDS-805. Block token: Client api changes for block token. Contributed by Ajay Kumar. 2019-01-16 13:38:27 -07:00
Xiaoyu Yao 2b115222cd HDDS-929. Remove ozone.max.key.len property. Contributed by Ajay Kumar. 2019-01-16 01:42:08 -07:00
Ajay Kumar f894d86b2f HDDS-115. GRPC: Support secure gRPC endpoint with mTLS. Contributed by Xiaoyu Yao. 2019-01-16 01:42:08 -07:00
Xiaoyu Yao 417951ab58 HDDS-884. Fix merge issue that causes NPE OzoneManager#httpServer. Contributed by Xiaoyu Yao. 2019-01-16 01:22:44 -07:00
Ajay Kumar 6d522dc05d HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar. 2019-01-16 01:22:43 -07:00
Anu Engineer 0c8829a9a1 HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate. 
Contributed by Anu Engineer.
 2019-01-16 01:16:25 -07:00
Ajay Kumar 825310608b HDDS-873. Fix TestSecureOzoneContainer NPE after HDDS-837. Contributed by Xiaoyu Yao. 2019-01-16 01:16:25 -07:00
Xiaoyu Yao 7e2770699c HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao. 2019-01-16 01:10:08 -07:00
Ajay Kumar bb4a26ca32 HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar. 2019-01-16 01:06:59 -07:00
Xiaoyu Yao 6d6b1a00c2 HDDS-836. Add TokenIdentifier Ozone for delegation token and block token. Contributed by Ajay Kumar. 2019-01-15 23:01:04 -07:00
Xiaoyu Yao 53120e2e6c HDDS-592. Fix ozone-secure.robot test. Contributed by Ajay Kumar. 2019-01-15 23:01:03 -07:00
Ajay Kumar a28ad7ad81 HDDS-753. SCM security protocol server is not starting. Contributed by Ajay Kumar. 2019-01-15 23:01:03 -07:00
Xiaoyu Yao 61e85d7cd1 HDDS-684. Fix HDDS-4 branch after HDDS-490 and HADOOP-15832. Contributed by Xiaoyu Yao. 2019-01-15 23:01:03 -07:00
Xiaoyu Yao 16e0bb874d HDDS-591. Adding ASF license header to kadm5.acl. Contributed by Ajay Kumar. 2019-01-15 23:01:03 -07:00
Xiaoyu Yao 0aab74026e HDDS-10. Add kdc docker image for secure ozone cluster. Contributed by Ajay Kumar. 2019-01-15 23:01:03 -07:00
Ajay Kumar d4511889fd HDDS-566. Move OzoneSecure docker-compose after HDDS-447. Contributed by Xiaoyu Yao. 2019-01-15 23:01:03 -07:00
Ajay Kumar 9920506b3d HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao. 2019-01-15 23:01:02 -07:00
Xiaoyu Yao 570b503e3e Fix merge conflicts 2019-01-15 22:54:50 -07:00
Xiaoyu Yao faf53f8262 HDDS-100. SCM CA: generate public/private key pair for SCM/OM/DNs. Contributed by Ajay Kumar. 2019-01-15 22:53:56 -07:00
Xiaoyu Yao e47135d9d9 HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar. 2019-01-15 22:53:55 -07:00
Xiaoyu Yao 914e93beeb HDDS-7. Enable kerberos auth for Ozone client in hadoop rpc. Contributed by Ajay Kumar. 2019-01-15 22:44:45 -07:00
Xiaoyu Yao d3920ecbb7 HDDS-5. Enable OzoneManager kerberos auth. Contributed by Ajay Kumar. 2019-01-15 22:40:08 -07:00
Xiaoyu Yao ff61931f91 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. 2019-01-15 22:16:58 -07:00
Márton Elek 05c84ab01c HDDS-977. Exclude dependency-reduced-pom.xml from ozone rat check. Contributed by Elek, Marton. 2019-01-14 16:30:21 -08:00
Hanisha Koneru a4eefe5765 HDDS-947. Implement OzoneManager State Machine. 2019-01-11 10:06:05 -08:00
Bharat Viswanadham 2091d1a4af HDDS-941. Rename ChunkGroupInputStream to keyInputStream and ChunkInputStream to BlockInputStream. Contributed by Shashikant Banerjee. 2019-01-09 20:02:36 -08:00
Hanisha Koneru f4617c6239 Revert "HDDS-947. Implement OzoneManager State Machine." 
This reverts commit 8dd11a1577.
 2019-01-09 11:20:57 -08:00
Hanisha Koneru 8dd11a1577 HDDS-947. Implement OzoneManager State Machine. 2019-01-09 10:57:24 -08:00
Márton Elek 32cf0412ce HDDS-965. Ozone: checkstyle improvements and code quality scripts. Contributed by Elek, Marton. 2019-01-08 22:27:19 +01:00
Márton Elek 999f31fd20 HDDS-924. MultipartUpload: S3 APi for complete Multipart Upload. Contributed by Bharat Viswanadham. 2019-01-08 15:48:48 +01:00
Márton Elek 695e93c2bb HDDS-969. Fix TestOzoneManagerRatisServer test failure. Contributed by Bharat Viswanadham. 2019-01-08 15:12:08 +01:00
Márton Elek 396ffba1aa HDDS-968. Fix TestObjectPut failures. Contributed by Bharat Viswanadham. 2019-01-08 14:54:38 +01:00
Bharat Viswanadham 4297e20c8b HDDS-926. Use Timeout rule for the the test methods in TestOzoneManager. Contributed by Dinesh Chitlangia. 2019-01-07 22:04:06 -08:00
Anu Engineer 06279ecc55 HDDS-946. AuditParser - insert audit to database in batches. 
contributed by Dinesh Chitlangia.
 2019-01-07 11:19:42 -08:00
Bharat Viswanadham cdfbec47ce HDDS-930. Multipart Upload: Abort multiupload request. Contributed by Bharat Viswanadham. 2019-01-07 11:09:14 -08:00
Márton Elek d66925a315 HDDS-902. MultipartUpload: S3 API for uploading a part file. Contributed by Bharat Viswanadham. 2019-01-07 12:39:46 +01:00
Márton Elek 992dd9d189 HDDS-901. MultipartUpload: S3 API for Initiate multipart upload. Contributed by Bharat Viswanadham. 2019-01-07 12:06:22 +01:00
Márton Elek d14c56d150 HDDS-916. MultipartUpload: Complete Multipart upload request. Contributed by Bharat Viswanadham. 2019-01-07 11:27:36 +01:00
Anu Engineer 999da98d67 HDDS-915. Submit client request to OM Ratis server. 
Contributed by Hanisha Koneru.
 2019-01-04 15:25:26 -08:00
Anu Engineer ddc0a40507 HDDS-896. Handle over replicated containers in SCM. 
Contributed by Nandakumar.
 2019-01-04 09:04:10 -08:00
Anu Engineer 14d232c0fe HDDS-957. Replace incorrect use of system property user.name. 
Contributed by Dinesh Chitlangia.
 2019-01-03 09:43:26 -08:00
Anu Engineer 996ab4874a HDDS-393. Audit Parser tool for processing ozone audit logs. 
Contributed by Dinesh Chitlangia.
 2019-01-02 11:52:12 -08:00
Nanda kumar 4d50ad030c HDDS-949. Update Ratis to 0.4.0-a8c4ca0-SNAPSHOT. 
Contributed by Xiaoyu Yao.
 2019-01-02 20:00:40 +05:30
Bharat Viswanadham 26e4be7022 HDDS-921. Add JVM pause monitor to Ozone Daemons (OM, SCM and Datanodes). Contributed by Bharat Viswanadham. 2018-12-23 17:33:25 -08:00
Bharat Viswanadham 13d3f99b37 HDDS-881.009. Encapsulate all client to OM requests into one request message. Contributed by Hanisha Koneru. 2018-12-20 09:12:42 -08:00
Nanda kumar ea621fa087 HDDS-934. Freon run hung on non-functional pipeline. 
Contributed by Nanda kumar.
 2018-12-20 11:13:15 +05:30
Arpit Agarwal b1ce9aa3b3 HDDS-940. Remove dead store to local variable in OmMetadataManagerImpl. Contributed by Dinesh Chitlangia. 2018-12-19 12:30:40 +05:30
Bharat Viswanadham 2f6bc25044 HDDS-911. Make TestOzoneManager unit tests independent. Contributed by Elek Marton. 2018-12-18 10:52:29 -08:00
Bharat Viswanadham 99b5061265 HDDS-914. Add Grafana support to ozoneperf docker container. Contributed by Dinesh Chitlangia. 2018-12-18 09:59:13 -08:00
Shashikant Banerjee b51e9e4317 HDDS-912. Update ozone to latest ratis snapshot build (0.4.0-3b0be02-SNAPSHOT). Contributed by Lokesh Jain. 2018-12-18 18:21:17 +05:30
Shashikant Banerjee 4ff1c46d95 HDDS-925. Rename ChunkGroupOutputStream to KeyOutputStream and ChunkOutputStream to BlockOutputStream. Contributed by Shashikant Banerjee. 2018-12-18 18:03:46 +05:30
Shashikant Banerjee ee10ba26dd HDDS-539. Ozone datanode command ignores the invalid options. Contributed by Vinicius Higa Murakami. 2018-12-18 16:10:27 +05:30
Xiaoyu Yao 94b368f29f HDDS-99. Adding SCM Audit log. Contributed by Dinesh Chitlangia. 2018-12-17 15:40:22 -08:00
Anu Engineer 5426653819 HDDS-908: NPE in TestOzoneRpcClient. 
Contributed by Ajay Kumar.
 2018-12-17 12:18:47 -08:00
Mukul Kumar Singh 92d44b2ad0 HDDS-876. Add blockade tests for flaky network. Contributed by Nilotpal Nandi. 2018-12-13 13:35:40 +05:30
Márton Elek d44b37d7ac HDDS-889. MultipartUpload: Support uploading a part file in ozone. Contributed by Bharat Viswanadham. 2018-12-12 14:16:21 +01:00
Márton Elek 21c9080121 HDDS-872. Add Dockerfile and skaffold config to deploy ozone dev build to k8s. Contributed by Elek, Marton. 2018-12-11 16:31:33 +01:00
Márton Elek 17a8708039 HDDS-879. MultipartUpload: Add InitiateMultipartUpload in ozone. Contributed by Bharat Viswanadham. 2018-12-10 10:05:38 +01:00
Jitendra Pandey 1afba83f2c HDDS-870. Avoid creating block sized buffer in ChunkGroupOutputStream. Contributed by Shashikant Banerjee. 2018-12-08 09:23:10 -08:00
Anu Engineer 5e773efd78 HDDS-909 : Default implementation for Ozone acls. 
Contributed by Ajay Kumar.
 2018-12-07 23:20:19 -08:00
Bharat Viswanadham 6c852f2a37 HDDS-892. Parse aws v2 headers without spaces in Ozone s3 gateway. Contributed by Elek Marton. 2018-12-06 15:37:34 -08:00
Hanisha Koneru 019836b113 HDDS-858. Start a Standalone Ratis Server on OM 2018-12-06 13:33:52 -08:00
Ajay Kumar 8d882c3786 HDDS-880. Create api for ACL handling in Ozone. (Contributed by Ajay Kumar) 2018-12-06 13:27:28 -08:00
Bharat Viswanadham 343aaea2d1 HDDS-864. Use strongly typed codec implementations for the tables of the OmMetadataManager. Contributed by Elek Marton. 2018-12-06 11:39:59 -08:00
Mukul Kumar Singh 7274115d57 HDDS-890. Handle OverlappingFileLockException during writeStateMachineData in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-12-04 20:12:35 +05:30
Bharat Viswanadham de4255509a HDDS-894. Content-length should be set for ozone s3 ranged download. Contributed by Elek Marton. 2018-12-03 15:17:44 -08:00
Bharat Viswanadham ef3b03b75a HDDS-885. Fix test failures due to ChecksumData. Contributed by Hanisha Koneru. 2018-12-03 10:52:49 -08:00
Yiqun Lin 3044b78bd0 HDDS-848. Create SCM metrics related to container state. Contributed by Bharat Viswanadham. 2018-12-03 17:16:34 +08:00
Shashikant Banerjee 8f3e12ff07 HDDS-882. Provide a config to optionally turn on/off the sync flag during chunk writes. Contributed by Shashikant Banerjee. 2018-12-02 08:06:24 +05:30
Shashikant Banerjee 5a3c7714c4 HDDS-887. Add DispatcherContext info to Dispatcher from containerStateMachine. Contributed by Shashikant Banerjee. 2018-12-02 08:00:35 +05:30
Bharat Viswanadham d15dc43659 HDDS-748. Use strongly typed metadata Table implementation. Contributed by Elek Marton. 2018-12-01 16:52:23 -08:00
Xiaoyu Yao ae5fbdd9ed HDDS-877. Ensure correct surefire version for Ozone test. Contributed by Xiaoyu Yao. 2018-11-29 11:37:36 -08:00
Shashikant Banerjee 5e102f9aa5 HDDS-850. ReadStateMachineData hits OverlappingFileLockException in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-11-29 22:20:08 +05:30
Ajay Kumar 184cced513 HDDS-808. Simplify OMAction and DNAction classes used for AuditLogging. Contributed by Dinesh Chitlangia. 2018-11-29 08:35:20 -08:00
Hanisha Koneru 64a4b6b08b HDDS-284. Checksum for ChunksData. 2018-11-28 10:53:12 -08:00
Anu Engineer 34a914be03 HDDS-846. Exports ozone metrics to prometheus. 
Contributed by Elek, Marton.
 2018-11-27 15:57:24 -08:00
Márton Elek 96c104d365 HDDS-766. Ozone shell create volume fails if volume name does not have a leading slash. Contributed by chencan. 2018-11-27 12:32:54 +01:00
Márton Elek 753f421350 HDDS-820. Use more strict data format for the Last-Modified headers of s3 gateway. Contributed by Elek, Marton. 2018-11-23 10:05:12 +01:00
Shashikant Banerjee ee44b069c6 HDDS-866. Handle RaftRetryFailureException in OzoneClient. Contributed by Shashikant Banerjee. 2018-11-22 15:02:07 +05:30
Ajay Kumar 8d2789c5eb HDDS-795. RocksDb specific classes leak from DBStore/Table interfaces. Contributed by Márton Elek. 2018-11-21 12:25:41 -08:00
Bharat Viswanadham c4d0ef6946 HDDS-816. Create OM metrics for bucket, volume, keys. Contributed by Bharat Viswanadham. 2018-11-21 10:35:39 -08:00
Márton Elek 34b6fa7d6b HDDS-732. Add read method which takes offset and length in SignedChunkInputStream. Contributed by chencan. 2018-11-21 17:59:36 +01:00
Márton Elek 22867deffa HDDS-791. Support Range header for ozone s3 object download. Contributed by Bharat Viswanadham. 2018-11-21 17:04:53 +01:00
Shashikant Banerjee c8b3dfa625 HDDS-860. Fix TestDataValidate unit tests. Contributed by Shashikant Banerjee. 2018-11-21 11:28:20 +05:30
Shashikant Banerjee 14e1a0a3d6 HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee. 2018-11-21 10:51:50 +05:30
Xiaoyu Yao f994b526a0 HDDS-855. Move OMMetadataManager from hadoop-ozone/ozone-manager to hadoop-ozone/common. Contributed by Ajay Kumar. 2018-11-20 20:31:07 -08:00
Shashikant Banerjee 1734ace35f Revert "HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee." 
This reverts commit c74783036d.
 2018-11-21 01:07:38 +05:30
Shashikant Banerjee c74783036d HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee. 2018-11-20 23:01:02 +05:30
Márton Elek 4ecdcc9620 HDDS-692. Use the ProgressBar class in the RandomKeyGenerator freon test. Contributed by Zsolt Horvath. 2018-11-20 15:24:20 +01:00
Márton Elek c946f1b121 Revert "HDDS-692. Use the ProgressBar class in the RandomKeyGenerator freon test. Contributed by Zsolt Horvath." 
This reverts commit 2a7f485991.
 2018-11-20 13:35:40 +01:00
Márton Elek 2a7f485991 HDDS-692. Use the ProgressBar class in the RandomKeyGenerator freon test. Contributed by Zsolt Horvath. 2018-11-20 12:57:27 +01:00
Yiqun Lin d0cc679441 HDDS-817. Create SCM metrics for disk from node report. Contributed by Bharat Viswanadham. 2018-11-20 14:22:30 +08:00
Mukul Kumar Singh b5d7b292c9 HDDS-718. Introduce new SCM Commands to list and close Pipelines. Contributed by Lokesh Jain. 2018-11-19 22:59:30 +05:30
Nanda kumar 93666087bc HDDS-847. TestBlockDeletion is failing. 
Contributed by Lokesh Jain.
 2018-11-19 17:04:18 +05:30
Nanda kumar 5a7ca6ac3e HDDS-837. Persist originNodeId as part of .container file in datanode. 
Contributed by Nanda kumar.
 2018-11-19 14:39:55 +05:30
Mukul Kumar Singh cfb915f3df HDDS-844. Add logic for pipeline teardown after timeout. Contributed by Lokesh Jain. 2018-11-19 12:11:05 +05:30
Xiaoyu Yao 29374999b6 HDDS-120. Adding HDDS datanode Audit Log. Contributed by Dinesh Chitlangia. 2018-11-16 14:59:59 -08:00
Nanda kumar c4d9640028 HDDS-801. Quasi close the container when close is not executed via Ratis. 
Contributed by Nanda kumar.
 2018-11-16 23:11:33 +05:30
Anu Engineer a16aa2f60b HDDS-825. Code cleanup based on messages from ErrorProne. 
Contributed by Anu Engineer.
 2018-11-15 17:36:09 -08:00
Bharat Viswanadham 57866b366f HDDS-821. Handle empty x-amz-storage-class header in Ozone S3 gateway. Contributed by Elek Marton. 2018-11-15 14:21:42 -08:00
Márton Elek d40912b1bd HDDS-223. Create acceptance test for using datanode plugin. Contributed by Sandeep Nemuri. 2018-11-15 15:18:07 +01:00
Márton Elek b12e69475b HDDS-658. Implement s3 bucket list backend call and use it from rest endpoint. Contributed by Bharat Viswanadham. 2018-11-15 12:22:16 +01:00
Anu Engineer 9b01f039eb HDDS-832. Docs folder is missing from the Ozone distribution package. Contributed by Elek, Marton. 2018-11-15 11:08:48 +01:00
Arpit Agarwal bac8807c8b HDDS-819. Match OzoneFileSystem behavior with S3AFileSystem. Contributed by Hanisha Koneru. 2018-11-14 16:12:06 -08:00
Shashikant Banerjee b57cc73f83 HDDS-774. Remove OpenContainerBlockMap from datanode. Contributed by Shashikant Banerjee. 2018-11-14 20:05:56 +05:30
Shashikant Banerjee 671fd6524b HDDS-675. Add blocking buffer and use watchApi for flush/close in OzoneClient. Contributed by Shashikant Banerjee. 2018-11-13 23:39:14 +05:30
Yiqun Lin f8713f8ade HDDS-831. TestOzoneShell in integration-test is flaky. Contributed by Nanda kumar. 2018-11-13 10:38:27 +08:00
Jitendra Pandey f944f33832 HDDS-709. Modify Close Container handling sequence on datanodes. Contributed by Shashikant Banerjee. 2018-11-12 14:08:39 -08:00
Nanda kumar 18fe65d756 HDDS-576. Move ContainerWithPipeline creation to RPC endpoint. 
Contributed by Nanda kumar.
 2018-11-12 23:33:11 +05:30
Márton Elek 9c32b50d61 HDDS-767. OM should not search for STDOUT root logger for audit logging. Contributed by Dinesh Chitlangia. 2018-11-12 10:54:41 +01:00
Tsz Wo Nicholas Sze 298d2502b0 HDDS-826. Update Ratis to 0.3.0-6f3419a-SNAPSHOT. 2018-11-09 18:35:40 -08:00
Nanda kumar 9fe50b4991 HDDS-733. Create container if not exist, as part of chunk write. 
Contributed by Lokesh Jain.
 2018-11-10 00:08:17 +05:30
Bharat Viswanadham 47194fefdd HDDS-823. OzoneRestClient is failing with NPE on getKeyDetails call. Contributed by Bharat Viswanadham. 2018-11-08 21:28:04 -08:00
Shashikant Banerjee 31614bcc7c HDDS-806. Update Ratis to latest snapshot version in ozone. Contributed by Tsz Wo Nicholas Sze and Mukul Kumar Singh. 2018-11-09 00:05:45 +05:30
Yiqun Lin 9317a61f3c HDDS-802. Container State Manager should get open pipelines for allocating container. Contributed by Lokesh Jain. 2018-11-08 23:41:43 +08:00
Nanda kumar c80f753b0e HDDS-737. Introduce Incremental Container Report. 
Contributed by Nanda kumar.
 2018-11-08 18:33:38 +05:30
Yiqun Lin addec29297 HDDS-809. Refactor SCMChillModeManager. 2018-11-07 13:54:08 +08:00
Anu Engineer 00a67f770e HDDS-810. Move the "\r\n" at the chunk end in TestSignedChunksInputStream#singlechunkwithoutend. 
Contributed by Chencan.
 2018-11-06 10:42:19 -08:00
Márton Elek 3d5cc1138a HDDS-798. Storage-class is showing incorrectly. Contributed by Bharat Viswanadham. 2018-11-06 18:28:54 +01:00
Akira Ajisaka 49412a1289
HDDS-811. [JDK 11] Fix compilation failures with jdk 11. Contributed by Lokesh Jain. 2018-11-06 21:26:43 +09:00
Shashikant Banerjee 942693bddd HDDS-799. Avoid ByteString to byte array conversion cost by using ByteBuffer in Datanode. Contributed by Mukul Kumar Singh. 2018-11-05 23:43:22 +05:30
Arpit Agarwal c8ca1747c0 HDDS-797. If DN is started before SCM, it does not register. Contributed by Hanisha Koneru. 2018-11-05 09:40:00 -08:00
Bharat Viswanadham 6d9c18cfa9 HDDS-788. Change title page of bucket browser in S3gateway. Contributed by Bharat Viswanadham. 2018-11-02 08:00:18 -07:00
Yiqun Lin d16d5f730e HDDS-751. Replace usage of Guava Optional with Java Optional. 2018-11-02 10:50:32 +08:00
Shashikant Banerjee e0ac3081e9 HDDS-771. ChunkGroupOutputStream stream entries need to be properly updated on closed container exception. Contributed by Lokesh Jain. 2018-11-01 15:43:48 +05:30
Shashikant Banerjee b13c56742a HDDS-697. update and validate the BCSID for PutSmallFile/GetSmallFile command. Contributed by Shashikant Banerjee. 2018-11-01 10:21:39 +05:30
Bharat Viswanadham c5eb237e3e HDDS-777. Fix missing jenkins issue in s3gateway module. Contributed by Bharat Viswanadham. 2018-10-31 19:11:12 -07:00
Bharat Viswanadham 6668c19daf HDDS-677. Create documentation for s3 gateway to the docs. Contributed by Elek Marton. 2018-10-31 11:48:23 -07:00
Arpit Agarwal 08bb0362e0 HDDS-759. Create config settings for SCM and OM DB directories. Contributed by Arpit Agarwal. 2018-10-31 11:23:15 -07:00
Bharat Viswanadham 478b2cba0d HDDS-773. Loading ozone s3 bucket browser could be failed. Contributed by Elek Marton. 2018-10-31 07:54:23 -07:00
Márton Elek b519f3f2a0 HDDS-659. Implement pagination in GET bucket (object list) endpoint. Contributed by Bharat Viswanadham. 2018-10-31 13:29:01 +01:00
Márton Elek ecac351aac HDDS-712. Use x-amz-storage-class to specify replication type and replication factor. Contributed by Bharat Viswanadham. 2018-10-31 13:28:59 +01:00
Nanda kumar e4f22b08e0 HDDS-755. ContainerInfo and ContainerReplica protobuf changes. 
Contributed by Nanda kumar.
 2018-10-31 10:29:35 +05:30
Anu Engineer 773f0d1519 HDDS-754. VolumeInfo#getScmUsed throws NPE. 
Contributed by Hanisha Koneru.
 2018-10-30 19:17:57 -07:00
Anu Engineer e33b61f335 HDDS-762. Fix unit test failure for TestContainerSQLCli & TestSCMMetrics. 
Contributed by Mukul Kumar Singh.
 2018-10-30 19:16:52 -07:00
Shashikant Banerjee 7757331dbc HDDS-749. Restructure BlockId class in Ozone. Contributed by Shashikant Banerjee. 2018-10-30 14:15:27 +05:30
Arpit Agarwal 496f0ffe90 HDDS-620. ozone.scm.client.address should be an optional setting. Contributed by chencan and Arpit Agarwal. 2018-10-29 17:14:18 -07:00