Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,181 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

25181 Commits

Author SHA1 Message Date
Siyao Meng 6a05d93de5 Fix TestFileStatusWithDefaultECPolicy. 
Change-Id: I5980792c534bdbdafab787552c467cf45e9336fe
 2021-08-19 22:28:19 -07:00
Siyao Meng 991fd8d746 Fix TestViewFileSystemHdfs. 
Change-Id: I026e35eecf45a5f2a0c0ba3c59cfea67a57d1125
 2021-08-19 22:26:15 -07:00
Siyao Meng 28dbf672c5 [Check static addition] Fix TestFSMainOperationsWebHdfs. 
Change-Id: Id3cd5178cd4b6a212a585f358a489cc50904b548
 2021-08-19 22:20:40 -07:00
Siyao Meng 52a0ca301b Fix TestErasureCodingPolicyWithSnapshot. 
Change-Id: I8169990a55f8aaa4429cede4d571ad0e3f489a0d
 2021-08-19 22:20:26 -07:00
Siyao Meng d5adaac439 Fix TestShortCircuitCache. 
Change-Id: Ie6fcec5dde5e19406e243df3bd88b232a5e49d06
 2021-08-19 22:20:26 -07:00
Siyao Meng 93e3046bf0 Fix TestScrLazyPersistFiles. 
Change-Id: I670fcb9a6429b6011f6ba13b8f716696d064655a
 2021-08-19 22:20:26 -07:00
Siyao Meng d11470c05e [Check line removal] Fix TestDFSClientFailover. 
Change-Id: Idb57ccbee96be77d6bb2df5ec5e3f2d25e846894
 2021-08-19 22:20:26 -07:00
Siyao Meng 800010489a Fix TestBlockReaderLocal. 
Change-Id: I3edfec1a5ed087a754ab36e58e4fe901ec6d2345
 2021-08-19 17:32:27 -07:00
Siyao Meng fa4dcdcf84 Fix TestDistributedFileSystem. 
Change-Id: I96b1cc10371a6a04d026c4bdd2d1f0ff1971714a
 2021-08-19 17:31:37 -07:00
Siyao Meng 8ba43d9210 Fix TestDiskBalancerRPC. 
Change-Id: Ib55211d6dd15615ff072ec8ba395b7449f0727a2
 2021-08-19 17:31:01 -07:00
Siyao Meng 8eeb645af4 Fix TestHDFSContractPathHandle. 
Change-Id: I23bf20e1a6ec8f864d5f250335d6b70debc71a44
 2021-08-19 17:31:01 -07:00
Akira Ajisaka a8b5d1d648 Test PR: Ran rewrite plugin 2021-08-14 23:00:44 +09:00
Renukaprasad C b90389ae98
HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-14 19:55:32 +09:00
Gautham B A 77383a4d55
HDFS-15976. Make mkdtemp cross platform (#2908) 2021-08-10 09:23:28 -07:00
Viraj Jasani 6a7883431f
HADOOP-17841. Remove ListenerHandle from Hadoop registry (#3278) 2021-08-09 16:57:53 +08:00
jianghuazhu 0c7b951e03
HDFS-16151. Improve the parameter comments related to ProtobufRpcEngine2#Server(). (#3256) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-08 14:55:55 +09:00
Akira Ajisaka 3565c9477d
HADOOP-17370. Upgrade commons-compress to 1.21 (#3274) 2021-08-08 11:23:28 +09:00
Akira Ajisaka 4972e7a246
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 2021-08-07 11:27:25 +09:00
Viraj Jasani 23e2a0b202
HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (#3266) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-07 11:20:35 +09:00
Gautham B A 4fd97e01e5
HADOOP-17787. Refactor fetching of credentials in Jenkins (#3167) 2021-08-06 11:56:27 -07:00
Bryan Beaudreault b0b867e977
HADOOP-17837: Add unresolved endpoint value to UnknownHostException (ADDENDUM) (#3276) 2021-08-06 21:54:07 +05:30
wangzhaohui e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-06 18:23:00 +08:00
Bryan Beaudreault 5e54d92e6e
HADOOP-17837: Add unresolved endpoint value to UnknownHostException (#3272) 2021-08-06 17:00:20 +08:00
wangzhaohui a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 2021-08-06 09:27:05 +01:00
Viraj Jasani 9fe1f24ec1
HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 2021-08-06 09:27:30 +08:00
Gautham B A e80b5a08e2
HADOOP-17816. Run optional CI for changes in C (#3238) 
* HADOOP-17816. Run optional CI for changes in C

* We need to ensure that we run
  the CI for all the platforms
  when there are changes in C files.

* Trigger CI

* Revert "Trigger CI"

This reverts commit ecb88350b9.
 2021-08-05 12:13:33 +00:00
Mehakmeet Singh 8d6a686953
HADOOP-17823. S3A S3Guard tests to skip if S3-CSE are enabled (#3263) 
Follow on to
* HADOOP-13887. Encrypt S3A data client-side with AWS SDK (S3-CSE)
* HADOOP-17817. S3A to raise IOE if both S3-CSE and S3Guard enabled

If the S3A bucket is set up to use S3-CSE encryption, all tests which turn
on S3Guard are skipped, so they don't raise any exceptions about
incompatible configurations.

Contributed by: Mehakmeet Singh
 2021-08-05 11:46:17 +01:00
Steve Loughran a67a0fd37a
YARN-10878. move TestNMSimulator off com.google (#3268) 
Converting from a class to a lambda-expression removes all need to reference the google stuff

Contributed by Steve Loughran
 2021-08-05 11:34:10 +01:00
sumangala-patki 3450522c2f
HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845) 
Contributed by Sumangala Patki
 2021-08-04 19:45:57 +01:00
Gergely Pollak 44bab51be4 YARN-10849 Clarify testcase documentation for TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by Szilard Nemeth 2021-08-04 15:43:57 +02:00
Tamas Domok 0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-04 15:27:46 +02:00
zhangshuyan0 10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-04 00:22:21 +08:00
Tamas Domok a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-03 16:19:04 +02:00
jianghuazhu 8616591b0c
HDFS-16149.Improve the parameter annotation in FairCallQueue#priorityLevels. (#3255) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2021-08-03 16:53:24 +08:00
Viraj Jasani ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
Steve Loughran 4627e9c7ef
HADOOP-17822. fs.s3a.acl.default not working after S3A Audit feature (#3249) 
Fixes the regression caused by HADOOP-17511 by moving where the
option  fs.s3a.acl.default is read -doing it before the RequestFactory
is created.

Adds

* A unit test in TestRequestFactory to verify the ACLs are set
  on all file write operations.
* A new ITestS3ACannedACLs test which verifies that ACLs really
  do get all the way through.
* S3A Assumed Role delegation tokens to include the IAM permission
  s3:PutObjectAcl in the generated role.

Contributed by Steve Loughran
 2021-08-02 15:26:56 +01:00
Steve Loughran ee466d4b40
HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240) 
This patch cuts down the size of directory trees used for
distcp contract tests against object stores, so making
them much faster against distant/slow stores.

On abfs, the test only runs with -Dscale (as was the case for s3a already),
and has the larger scale test timeout.

After every test case, the FileSystem IOStatistics are logged,
to provide information about what IO is taking place and
what it's performance is.

There are some test cases which upload files of 1+ MiB; you can
increase the size of the upload in the option
"scale.test.distcp.file.size.kb" 
Set it to zero and the large file tests are skipped.

Contributed by Steve Loughran.
 2021-08-02 11:36:43 +01:00
zhuqi-lucas efb3fa2bf5 YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang. 2021-08-02 18:23:33 +08:00
jenny e31169c864
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup (#1808) 
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-02 15:36:53 +09:00
Bobby Wang 266b1bd1bb
HADOOP-17812. NPE in S3AInputStream read() after failure to reconnect to store (#3222) 
This improves error handling after multiple failures reading data
-when the read fails and attempts to reconnect() also fail.

Contributed by Bobby Wang.
 2021-07-30 20:04:11 +01:00
Petre Bogdan Stolojan a218038960
HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem (#3101) 
This work
* Defines the behavior of FileSystem.copyFromLocal in filesystem.md
* Implements a high performance implementation of copyFromLocalOperation
  for S3 
* Adds a contract test for the operation: AbstractContractCopyFromLocalTest
* Implements the contract tests for Local and S3A FileSystems

Contributed by: Bogdan Stolojan
 2021-07-30 19:42:08 +01:00
Wei-Chiu Chuang 6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-30 08:29:56 -07:00
Tamas Domok 798a0837c1
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty (#3206) 
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).

Any IO error would have resulted in the same fallback path.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-07-30 12:16:46 +02:00
Gautham B A 13467f4524
HADOOP-17815. Run CI for Centos 7 (#3231) 2021-07-29 10:50:57 -07:00
Benjamin Teke 8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 19:15:27 +02:00
Benjamin Teke ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 17:56:14 +02:00
Szilard Nemeth 74770c8a16 YARN-10663. Add runningApps stats in SLS. Contributed by Vadaga Ananyo Rao 2021-07-29 17:37:40 +02:00
Szilard Nemeth 54f9fff218 YARN-10628. Add node usage metrics in SLS. Contributed by Vadaga Ananyo Rao 2021-07-29 13:43:40 +02:00
minni31 d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194) 
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
 2021-07-29 14:55:39 +05:30
jianghuazhu 6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 21:57:28 -07:00