Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,294 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

25294 Commits

Author SHA1 Message Date
lzx404243 7b5be74228
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() (#2968) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-30 08:39:33 +09:00
Liang-Chi Hsieh 73a0c31370
HADOOP-17877. BuiltInGzipCompressor header and trailer should not be static variables (#3350) 2021-08-29 08:21:55 -07:00
Tamas Domok 16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-27 23:09:54 +02:00
jianghuazhu 4c94831364
HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-27 11:41:44 +08:00
Viraj Jasani aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-26 00:37:38 -07:00
LeonGao b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284) 
* Add DNS resolution for QJM

* Add log

* Resolve comments

* checkstyle

* typo
 2021-08-25 17:40:12 -07:00
jianghuazhu ad54f5195c
HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS. (#3307) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-08-25 17:34:45 +08:00
Viraj Jasani 1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-08-25 17:37:25 +09:00
Gautham B A c8e5864838
HADOOP-17854. Run junit in Jenkins only if surefire reports exist (#3319) 2021-08-24 10:26:26 -07:00
Viraj Jasani fc566ad9b0
HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization (#3321) 2021-08-24 22:41:59 +09:00
Szilard Nemeth e06a5cb197 YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke 2021-08-24 15:27:34 +02:00
Neil 9084c728eb
HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315) 2021-08-24 11:15:47 +08:00
Gautham B A b6d1971820
HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (#3311) 2021-08-20 10:04:55 -07:00
litao 07627ef19e
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-08-19 12:30:38 +08:00
litao 0e922aab41
HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291) 2021-08-19 09:51:00 +08:00
sumangala-patki dcddc6a59f
HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters (#2975) 2021-08-18 19:14:10 +05:30
Steve Loughran ee07b90286
HADOOP-17836. Improve logging on ABFS error reporting (#3281) 
Contributed by Steve Loughran.
 2021-08-18 11:39:17 +01:00
Gautham B A 409878ef80
HDFS-16174. Refactor TempFile and TempDir in libhdfs++ (#3303) 2021-08-17 09:49:52 -07:00
srinivasst 4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node (#3287) 
* Account for scheduled AM containers before deactivating node

* Move AM container check to separate method.

* Fix UTs

* Fix UTs

* Remove unnecessary import

* Add timeout for UT
 2021-08-17 14:18:55 +05:30
jianghuazhu 53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-17 16:47:10 +08:00
Liang-Chi Hsieh 6014a089fd
HADOOP-17825. Add BuiltInGzipCompressor (#3250) 
Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is not loaded. So, without Hadoop native codec installed, saving SequenceFile using GzipCodec will throw exception like "SequenceFile doesn't work with GzipCodec without native-hadoop code!"

Same as other codecs which we migrated to using prepared packages (lz4, snappy), it will be better if we support GzipCodec generally without Hadoop native codec installed. Similar to BuiltInGzipDecompressor, we can use Java Deflater to support BuiltInGzipCompressor.
 2021-08-16 10:08:03 -07:00
Viraj Jasani 6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-16 14:54:25 +09:00
Viraj Jasani 919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map (#3296) 2021-08-16 13:40:05 +08:00
Siyao Meng 3aaac8a1f6
HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-14 21:19:10 +09:00
Akira Ajisaka 3ec53a33ba
HADOOP-17799. Improve the GitHub pull request template (#3277) 
Reviewed-by: Yi Sheng <yisheng@apache.org>
 2021-08-14 21:16:15 +09:00
Renukaprasad C b90389ae98
HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-14 19:55:32 +09:00
Gautham B A 77383a4d55
HDFS-15976. Make mkdtemp cross platform (#2908) 2021-08-10 09:23:28 -07:00
Viraj Jasani 6a7883431f
HADOOP-17841. Remove ListenerHandle from Hadoop registry (#3278) 2021-08-09 16:57:53 +08:00
jianghuazhu 0c7b951e03
HDFS-16151. Improve the parameter comments related to ProtobufRpcEngine2#Server(). (#3256) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-08 14:55:55 +09:00
Akira Ajisaka 3565c9477d
HADOOP-17370. Upgrade commons-compress to 1.21 (#3274) 2021-08-08 11:23:28 +09:00
Akira Ajisaka 4972e7a246
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 2021-08-07 11:27:25 +09:00
Viraj Jasani 23e2a0b202
HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (#3266) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-07 11:20:35 +09:00
Gautham B A 4fd97e01e5
HADOOP-17787. Refactor fetching of credentials in Jenkins (#3167) 2021-08-06 11:56:27 -07:00
Bryan Beaudreault b0b867e977
HADOOP-17837: Add unresolved endpoint value to UnknownHostException (ADDENDUM) (#3276) 2021-08-06 21:54:07 +05:30
wangzhaohui e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-06 18:23:00 +08:00
Bryan Beaudreault 5e54d92e6e
HADOOP-17837: Add unresolved endpoint value to UnknownHostException (#3272) 2021-08-06 17:00:20 +08:00
wangzhaohui a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 2021-08-06 09:27:05 +01:00
Viraj Jasani 9fe1f24ec1
HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 2021-08-06 09:27:30 +08:00
Gautham B A e80b5a08e2
HADOOP-17816. Run optional CI for changes in C (#3238) 
* HADOOP-17816. Run optional CI for changes in C

* We need to ensure that we run
  the CI for all the platforms
  when there are changes in C files.

* Trigger CI

* Revert "Trigger CI"

This reverts commit ecb88350b9.
 2021-08-05 12:13:33 +00:00
Mehakmeet Singh 8d6a686953
HADOOP-17823. S3A S3Guard tests to skip if S3-CSE are enabled (#3263) 
Follow on to
* HADOOP-13887. Encrypt S3A data client-side with AWS SDK (S3-CSE)
* HADOOP-17817. S3A to raise IOE if both S3-CSE and S3Guard enabled

If the S3A bucket is set up to use S3-CSE encryption, all tests which turn
on S3Guard are skipped, so they don't raise any exceptions about
incompatible configurations.

Contributed by: Mehakmeet Singh
 2021-08-05 11:46:17 +01:00
Steve Loughran a67a0fd37a
YARN-10878. move TestNMSimulator off com.google (#3268) 
Converting from a class to a lambda-expression removes all need to reference the google stuff

Contributed by Steve Loughran
 2021-08-05 11:34:10 +01:00
sumangala-patki 3450522c2f
HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845) 
Contributed by Sumangala Patki
 2021-08-04 19:45:57 +01:00
Gergely Pollak 44bab51be4 YARN-10849 Clarify testcase documentation for TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by Szilard Nemeth 2021-08-04 15:43:57 +02:00
Tamas Domok 0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-04 15:27:46 +02:00
zhangshuyan0 10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-04 00:22:21 +08:00
Tamas Domok a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-03 16:19:04 +02:00
jianghuazhu 8616591b0c
HDFS-16149.Improve the parameter annotation in FairCallQueue#priorityLevels. (#3255) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2021-08-03 16:53:24 +08:00
Viraj Jasani ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
Steve Loughran 4627e9c7ef
HADOOP-17822. fs.s3a.acl.default not working after S3A Audit feature (#3249) 
Fixes the regression caused by HADOOP-17511 by moving where the
option  fs.s3a.acl.default is read -doing it before the RequestFactory
is created.

Adds

* A unit test in TestRequestFactory to verify the ACLs are set
  on all file write operations.
* A new ITestS3ACannedACLs test which verifies that ACLs really
  do get all the way through.
* S3A Assumed Role delegation tokens to include the IAM permission
  s3:PutObjectAcl in the generated role.

Contributed by Steve Loughran
 2021-08-02 15:26:56 +01:00
Steve Loughran ee466d4b40
HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240) 
This patch cuts down the size of directory trees used for
distcp contract tests against object stores, so making
them much faster against distant/slow stores.

On abfs, the test only runs with -Dscale (as was the case for s3a already),
and has the larger scale test timeout.

After every test case, the FileSystem IOStatistics are logged,
to provide information about what IO is taking place and
what it's performance is.

There are some test cases which upload files of 1+ MiB; you can
increase the size of the upload in the option
"scale.test.distcp.file.size.kb" 
Set it to zero and the large file tests are skipped.

Contributed by Steve Loughran.
 2021-08-02 11:36:43 +01:00