Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,658 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5744 Commits

Author SHA1 Message Date
Ayush Saxena d05655d2ad
Revert "HADOOP-18082.Add debug log when RPC#Reader gets a Call. (#3891). Contributed by JiangHua Zhu." 
Exposes a Race Condition. Which leads to test failures in YARN. (HADOOP-18143)

This reverts commit 2025243fbf.
 2022-02-28 21:44:24 +05:30
Owen O'Malley 12fa38d546
HADOOP-18139: Allow configuration of zookeeper server principal. 
Fixes #4024

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-02-24 15:01:50 -08:00
monthonk 1f157f802d
HADOOP-17386. Change default fs.s3a.buffer.dir to be under Yarn container path on yarn applications (#3908) 
Co-authored-by: Monthon Klongklaew <monthonk@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-22 13:50:27 +09:00
jianghuazhu 589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00
Steve Loughran cae749b076
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I73af19d2e2e41f4ba686c470726a80c3903a1950
 2022-02-21 17:08:56 +00:00
jianghuazhu 2025243fbf
HADOOP-18082.Add debug log when RPC#Reader gets a Call. (#3891). Contributed by JiangHua Zhu. 2022-02-17 01:49:45 +05:30
Chentao Yu 19d90e62fb HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 2022-02-15 15:58:24 -08:00
GuoPhilipse b68964336d
HDFS-16449. Fix hadoop web site release notes and changelog not available (#3967) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-14 05:38:28 +09:00
daimin 0e74f1e467
Fix thread safety of EC decoding during concurrent preads (#3881) 2022-02-11 10:20:00 +08:00
Xing Lin ca8ba24051 HADOOP-18110. ViewFileSystem: Add Support for Localized Trash Root 
Fixes #3956
 2022-02-10 16:43:04 -08:00
Steve Loughran efdec92cab
HADOOP-18091. S3A auditing leaks memory through ThreadLocal references (#3930) 
Adds a new map type WeakReferenceMap, which stores weak
references to values, and a WeakReferenceThreadMap subclass
to more closely resemble a thread local type, as it is a
map of threadId to value.

Construct it with a factory method and optional callback
for notification on loss and regeneration.

 WeakReferenceThreadMap<WrappingAuditSpan> activeSpan =
      new WeakReferenceThreadMap<>(
          (k) -> getUnbondedSpan(),
          this::noteSpanReferenceLost);

This is used in ActiveAuditManagerS3A for span tracking.

Relates to
* HADOOP-17511. Add an Audit plugin point for S3A
* HADOOP-18094. Disable S3A auditing by default.

Contributed by Steve Loughran.
 2022-02-10 12:31:41 +00:00
Abhishek Das 3684c7f66a
HADOOP-18100: Change scope of inner classes in InodeTree to make them accessible outside package 
Fixes #3950

Signed-off-by: Owen O'Malley <omalley@apache.org>
 2022-02-03 16:28:04 -08:00
Ayush Saxena aeae5716cc
Revert "HADOOP-18024. SocketChannel is not closed when IOException happens in Server$Listener.doAccept (#3719)" 
This reverts commit 6ed01585eb.

Breaks TestIPC#testIOEOnListenerAccept
 2022-02-01 14:11:25 +05:30
Li MingXiang e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. (#3900). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-27 16:53:21 +08:00
Viraj Jasani 4faac58841
HADOOP-18089. Test coverage for Async profiler servlets (#3913) 
Reviewed-by: Akira Ajisaka <akiraaj@amazon.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-01-26 11:24:16 +08:00
Xing Lin 0d17b629ff
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java (#3918). Contributed by Xing Lin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-01-25 19:40:18 +05:30
Steve Loughran 14ba19af06
HADOOP-17409. Remove s3guard from S3A module (#3534) 
Completely removes S3Guard support from the S3A codebase.

If the connector is configured to use any metastore other than
the null and local stores (i.e. DynamoDB is selected) the s3a client
will raise an exception and refuse to initialize.

This is to ensure that there is no mix of S3Guard enabled and disabled
deployments with the same configuration but different hadoop releases
-it must be turned off completely.

The "hadoop s3guard" command has been retained -but the supported
subcommands have been reduced to those which are not purely S3Guard
related: "bucket-info" and "uploads".

This is major change in terms of the number of files
changed; before cherry picking subsequent s3a patches into
older releases, this patch will probably need backporting
first.

Goodbye S3Guard, your work is done. Time to die.

Contributed by Steve Loughran.
 2022-01-17 18:08:57 +00:00
ahmarsuhail 7542677470
HADOOP-16223. Remove misleading fs.s3a.delegation.tokens.enabled prompt (#3879) 
Contributed by Ahmar Suhail
 2022-01-12 17:25:17 +00:00
Viraj Jasani 93294f0329
HADOOP-18077. ProfileOutputServlet unable to proceed due to NPE (#3875) 2022-01-12 16:20:34 +08:00
litao 39efbc6b6f
HDFS-16404. Fix typo for CachingGetSpaceUsed (#3844). Contributed by tomscut. 2022-01-09 16:41:10 +08:00
Viraj Jasani f64fda0f00
HADOOP-18055. Async Profiler endpoint for Hadoop daemons (#3824) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-06 17:56:49 +08:00
Mukund Thakur da0a6ba1ce
HADOOP-18065. ExecutorHelper.logThrowableFromAfterExecute() is too noisy. (#3860) 
Downgrading warn logs to debug in case of InterruptedException

Contributed By: Mukund Thakur
 2022-01-06 10:54:27 +05:30
jianghuazhu 7398a0f1b2
HADOOP-18063. Remove unused import AbstractJavaKeyStoreProvider in Shell class. (#3846) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 11:25:13 +09:00
jianghuazhu 43afd1753a
HDFS-16394.RPCMetrics increases the number of handlers in processing. (#3822) 2021-12-31 16:40:14 +08:00
Ashutosh Gupta caab29ec88
HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-28 21:44:38 +09:00
Viraj Jasani 04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
Dhananjay Badaya 4483607a4e
HADOOP-13500. Synchronizing iteration of Configuration properties object (#3775) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-17 16:05:46 +09:00
PHILO-HE 8e08f43e03
HDFS-16014: Fix an issue in checking native pmdk lib by 'hadoop checknative' command (#3762) 2021-12-14 14:45:12 +05:30
Wei-Chiu Chuang d7c5400fbc
HADOOP-17982. OpensslCipher initialization error should log a WARN message. (#3599) 
Change-Id: I070fc4784679b3be73aa3a11201bbae23c20ad4e
 2021-12-10 18:14:04 +09:00
Akira Ajisaka 9b9e2ef87f
HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:36:31 +09:00
Haoze Wu 6ed01585eb
HADOOP-18024. SocketChannel is not closed when IOException happens in Server$Listener.doAccept (#3719) 2021-12-08 18:48:43 +09:00
Andras Gyori 47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars (#3710) 
Contributed by Andras Gyori
 2021-12-03 16:44:58 +00:00
Desmond Sisson df4197592f
HADOOP-18029: Update CompressionCodecFactory to handle uppercase file extensions (#3739) 
Co-authored-by: Desmond Sisson <sissonde@amazon.com>
 2021-12-01 15:36:54 -08:00
smarthan 932a78fe38
HADOOP-18023. Allow cp command to run with multi threads. (#3721) 2021-11-29 12:45:08 +00:00
Takanobu Asanuma 9c887e5b82
HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-25 14:05:04 +09:00
huhaiyang 99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3708) 2021-11-25 10:20:42 +08:00
Steve Loughran 98fe0d0fc3
HADOOP-17979. Add Interface EtagSource to allow FileStatus subclasses to provide etags (#3633) 
Contributed by Steve Loughran
 2021-11-24 17:33:12 +00:00
Istvan Fajth ae3ba45db5
HADOOP-17975. Fallback to simple auth does not work for a secondary DistributedFileSystem instance. (#3579) 2021-11-24 10:44:57 +00:00
smarthan 63018dc73f
HADOOP-17998. Allow get command to run with multi threads. (#3645) 2021-11-22 11:37:05 +00:00
Hui Fei 8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630)" (#3697) 2021-11-22 10:05:21 +08:00
huhaiyang 91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630) 2021-11-17 22:41:06 +08:00
Abhishek Das 54a1d78e16 HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 2021-11-16 17:56:30 -08:00
litao c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-16 11:19:14 +09:00
Viraj Jasani e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-11-15 22:57:24 +09:00
litao 573b358fce
HADOOP-18003. Add a method appendIfAbsent for CallerContext (#3644) 2021-11-15 22:44:49 +09:00
litao 60acf8434d
HDFS-16319. Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount (#3653). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:10:24 +05:30
litao 7fb1bb8f35
HADOOP-18005. Correct log format for LdapGroupsMapping (#3647). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-12 20:12:18 +05:30
litao 359b03c83e
HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-04 10:16:03 +09:00
Steve Loughran 6c6d1b64d4
HADOOP-17928. Syncable: S3A to warn and downgrade (#3585) 
This switches the default behavior of S3A output streams
to warning that Syncable.hsync() or hflush() have been
called; it's not considered an error unless the defaults
are overridden.

This avoids breaking applications which call the APIs,
at the risk of people trying to use S3 as a safe store
of streamed data (HBase WALs, audit logs etc).

Contributed by Steve Loughran.
 2021-11-02 13:26:16 +00:00
Viraj Jasani 516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00