Commit Graph

22789 Commits

Author SHA1 Message Date
Billie Rinaldi 2e2e5401f2 YARN-9718. Fixed yarn.service.am.java.opts shell injection. Contributed by Eric Yang 2019-09-05 12:49:58 -07:00
Inigo Goiri 72d8b92ba5 HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun. 2019-09-05 11:44:02 -07:00
Steve Loughran 511df1e837 HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions
Contributed by Steve Loughran.

This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.

For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.

Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
2019-09-05 14:25:15 +01:00
Nanda kumar 2b16d5377c
HDDS-1898. GrpcReplicationService#download cannot replicate the container. (#1326) 2019-09-05 18:12:36 +05:30
Tao Yang 172bcd8e01 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. Contributed by zhuqi. 2019-09-05 16:20:05 +08:00
Hrishikesh Gadre f347c348d8 Add support for checksum verification in data scrubber
Signed-off-by: Anu Engineer <aengineer@apache.org>
2019-09-04 20:31:49 -07:00
Rohith Sharma K S 3db7184082 YARN-9804. Update ATSv2 document for latest feature supports. 2019-09-05 08:42:17 +05:30
Ayush Saxena 2f70b52a5b HDFS-14812. RBF: MountTableRefresherService should load cache when refresh. Contributed by xuzq. 2019-09-05 08:29:56 +05:30
Xiaoyu Yao ae287474c0
HDDS-2079. Fix TestSecureOzoneManager. Contributed by Xiaoyu Yao. (#1400) 2019-09-04 14:03:38 -07:00
Erik Krogen 337e9b794d HADOOP-16268. Allow StandbyException to be thrown as CallQueueOverflowException when RPC call queue is filled. Contributed by CR Hota. 2019-09-04 08:22:02 -07:00
Nanda kumar 1ae775975b
HDDS-2077. Add maven-gpg-plugin.version to pom.ozone.xml. (#1396) 2019-09-04 15:28:59 +05:30
Surendra Singh Lilhore 05704754a0 HDFS-14777. RBF: Set ReadOnly is failing for mount Table but actually readonly succeed to set. Contributed by Ranith Sardar. 2019-09-04 15:09:09 +05:30
Bharat Viswanadham cfa41a49af
HDDS-2018. Handle Set DtService of token for OM HA. (#1371) 2019-09-03 14:06:14 -07:00
Bharat Viswanadham f25fe92743
HDDS-1909. Use new HA code for Non-HA in OM. (#1225) 2019-09-03 13:24:32 -07:00
Xiaoyu Yao 3c117163a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 2019-09-03 09:29:58 -07:00
Ayush Saxena 91b01a1db7 HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena. 2019-09-03 18:08:42 +05:30
Sammi Chen 0f549ec02a HDDS-2069. Default values of properties hdds.datanode.storage.utilization.{critical | warning}.threshold are not reasonable. (#1393) 2019-09-03 17:58:48 +05:30
Nanda kumar ac5a0ae6d0
HADOOP-16534. Exclude submarine from hadoop source build. (#1356) 2019-09-03 17:40:38 +05:30
Nanda kumar 0b9704f610
HDDS-1810. SCM command to Activate and Deactivate pipelines. (#1224) 2019-09-03 16:50:57 +05:30
avijayanhwx b53d19a343 HDDS-1783 : Latency metric for applyTransaction in ContainerStateMachine (#1363). 2019-09-03 15:18:50 +05:30
Surendra Singh Lilhore 5ff76cb8bc HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 2019-09-03 12:37:09 +05:30
Zhankun Tang bb26514ba9 YARN-9785. Fix DominantResourceCalculator when one resource is zero. Contributed by Bibin A Chundatt, Sunil Govindan, Bilwa S T. 2019-09-03 14:55:15 +08:00
bibinchundatt 03489124ea YARN-9797. LeafQueue#activateApplications should use resourceCalculator#fitsIn. Contributed by Bilwa S T. 2019-09-03 11:53:34 +05:30
Stephen O'Donnell 915cbc91c0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-02 09:47:04 -07:00
Wei-Chiu Chuang d207aba026 Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell."
This reverts commit 7bebad61d9.
2019-09-02 09:43:44 -07:00
Ayush Saxena 040f6e93bb HDFS-14654. RBF: TestRouterRpc#testNamenodeMetrics is flaky. Contributed by Chen Zhang. 2019-09-02 13:01:52 +05:30
Abhishek Modi f4d6e82075 YARN-9400. Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId. Contributed by Prabhu Joseph. 2019-09-02 10:58:23 +05:30
Abhishek Modi b939ae98e6 YARN-8174. Add containerId to ResourceLocalizationService fetch failure log statement. Contributed by Prabhu Joseph. 2019-09-02 10:45:59 +05:30
Abhishek Modi bc27f7fb6d YARN-7982. Do ACLs check while retrieving entity-types per application. Contribued by Prabhu Joseph. 2019-09-02 10:15:59 +05:30
Ayush Saxena c3abfcefdd HDFS-13843. RBF: Add optional parameter -d for detailed listing of mount points. Contributed by Ayush Saxena. 2019-09-02 07:24:04 +05:30
Abhishek Modi 751b5a1ac8 YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph. 2019-09-01 22:40:40 +05:30
Ayush Saxena 18d74fe41c HDFS-14711. RBF: RBFMetrics throws NullPointerException if stateStore disabled. Contributed by Chen Zhang. 2019-09-01 12:53:10 +05:30
Abhishek Modi fef65b4c2b YARN-9793. Remove duplicate sentence from TimelineServiceV2.md. Contributed by Julia Kinga Marton. 2019-09-01 12:43:26 +05:30
Abhishek Modi c7ef4fb74a YARN-8678. Queue Management API - rephrase error messages. Contributed by Prabhu Joseph. 2019-09-01 10:00:04 +05:30
Abhishek Modi d2d963f3d4 YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 2019-09-01 09:54:46 +05:30
Márton Elek c187d2cb9f HDDS-2060. Create Ozone specific LICENSE file for the Ozone source package
Signed-off-by: Anu Engineer <aengineer@apache.org>
2019-08-31 15:01:46 -07:00
Abhishek Modi 7d998cb627 YARN-9800. TestRMDelegationTokens can fail in testRemoveExpiredMasterKeyInRMStateStore. Contributed by Adam Antal. 2019-08-31 10:03:09 +05:30
Abhishek Modi eeccd2f6f6 YARN-9798. ApplicationMasterServiceTestBase#testRepeatedFinishApplicationMaster fails intermittently. Contributed by Tao Yang. 2019-08-31 09:57:45 +05:30
Doroszlai, Attila a2d083f2c5 HDDS-1413. Attempt to fix TestCloseContainerCommandHandler by adjusting timeouts
Signed-off-by: Anu Engineer <aengineer@apache.org>
2019-08-30 18:17:08 -07:00
Doroszlai, Attila c4411f7fdf HDDS-2042. Avoid log on console with Ozone shell
Signed-off-by: Anu Engineer <aengineer@apache.org>
2019-08-30 17:40:34 -07:00
Doroszlai, Attila 70855126d1 HDDS-2061. Add hdds.container.chunk.persistdata as exception to TestOzoneConfigurationFields (#1382) 2019-08-30 12:31:52 -07:00
Doroszlai, Attila 472a26d2b8 HDDS-2063. Integration tests create untracked file audit.log (#1384) 2019-08-30 12:23:25 -07:00
dineshchitlangia 46696bd9b0 HDDS-2014. Create Symmetric Key for GDPR (#1362) 2019-08-30 09:55:36 -07:00
Xiaoyu Yao ec34cee5e3
HDDS-2047. Datanodes fail to come up after 10 retries in a secure env… (#1379) 2019-08-30 09:27:37 -07:00
Abhishek Modi c929b383f8 YARN-9540. TestRMAppTransitions fails intermittently. Contributed by Tao Yang. 2019-08-30 17:17:43 +05:30
Surendra Singh Lilhore 96f7dc1992 HDFS-14796. Define LOG instead of BlockManager.LOG in ErasureCodingWork/ReplicationWork. Contributed by Fei Hui. 2019-08-30 14:31:05 +05:30
Nanda kumar 22a58615a2
HDDS-2058. Remove hadoop dependencies in ozone build
Closes #1376
2019-08-30 10:21:04 +02:00
Rohith Sharma K S 5b557de721 YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby. Contributed by Tao Yang. 2019-08-30 10:35:09 +05:30
Brian Grunkemeyer d59fc59c9f HADOOP-15998. Ensure jar validation works on Windows.
* use a different path separator for artifacts to test
* fail if and commands fail (including the jar listing)
* handle different line endings from jar listing
* make sure we have bash 3.1+

Co-authored-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Abhishek Modi <abmodi@apache.org>
Signed-off-by: Rohith Sharma K S <rohithsharmaks@apache.org>
2019-08-29 23:09:04 -05:00
Ayush Saxena e220dac15c HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:40:13 +05:30