Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,506 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

9832 Commits

Author SHA1 Message Date
Steve Loughran ad024999d8
HADOOP-18487. change scope so that protobuf explicitly exported again 
rename protobuf2.scope to transient.protobuf2.scope as it is
only used in transient exports

Change-Id: Icfa54ad440834314ee69ec9bf3d585f981e20fef
 2023-03-06 10:20:46 +00:00
Steve Loughran f198ebc0cf
HADOOP-18487. protobuf -style and imports 
Change-Id: I6e33572b42ebd79ed1b8f81f20cc0e17794b0c4a
 2023-03-06 10:20:46 +00:00
Steve Loughran 40dd832153
HADOOP-18487. making protobuf 2.5 optional 
add a line between updated methods

Change-Id: Idb820bf1574a9749e3ed3f180e25081d68e6f0c0
 2023-03-06 10:20:46 +00:00
Steve Loughran 6c2197b645
HADOOP-18487. making protobuf 2.5 optional 
spotbugs and cleanup of uses of getRemoteException()

Change-Id: I5140094acc422a4df872ffa06fd944b7ea55ce9c
 2023-03-06 10:20:45 +00:00
Steve Loughran 5f86dd58bd
HADOOP-18487. making protobuf 2.5 optional 
* new org.apache.hadoop.ipc.internal for internal only classes
* with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* findbugs, protobuf source patch etc
* can spec new export policies for the protobuf jar in hadoop-common
  and other places it is referenced. hadoop-common back @compile

Change-Id: I61a99d2fd673259ab50d000f28a29e8a38aaf1b1
 2023-03-06 10:20:45 +00:00
Steve Loughran ce2b9d98cb
HADOOP-18487. protobuf 2.5 marked as provided. 
The methods which were overloaded with shaded/unshaded protobuf paramters
now have @Private ones with different names. the existing overloaded
methods are still there but deprecated, in case things use them.

* removed the extra imports needed before this change went in
* protobuf2.scope is the name for the scope var

+added a test for exception extraction, so yetus stops complaining

I am thinking of merging this in to 3.3.5 with the scope set to "compile",
as we get today. That way
* the overloading changes are in
* anyone who wants to cut their own release without protobuf2.5 can do it

Change-Id: I3423720c7047c63f7c9797d6e386774bff10b21a
 2023-03-06 10:20:45 +00:00
Steve Loughran 91acb8da23
HADOOP-18487. protobuf 2.5 marked as provided. 
The option protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* all declarations except those in yarn-csi are updated
* those modules which don't compile without their own explicit
  import (hadoop-hdfs-client, hadoop-hdfs-rbf)

It's actually interesting to see where/how that compile fails

hadoop-hdfs-client: ClientNamenodeProtocolTranslatorPB
hadoop-hdfs-rbf:RouterAdminProtocolTranslatorPB

both with "class file for com.google.protobuf.ServiceException not found",
even though *neither class uses it*

what they do have is references to ProtobufHelper.getRemoteException(),
which is overloaded to both the shaded ServiceException and the original one

Hypothesis: the javac overload resolution needs to look at the entire
class hierarchy before it can decide which one to use.

Proposed: add a new method
 ioe extractException(org.apache.hadoop.thirdparty.protobuf.ServiceException)
and move our own code to it. Without the overloading the classes should not
be needed

Change-Id: I70354abfe3f1fdc03c418dac88e60f8cc4929a33
 2023-03-06 10:20:45 +00:00
ZanderXu 6bd2444815
HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-03-01 16:18:38 -08:00
Tom 162288bc0a
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) 
HDFS-16896 clear ignoredNodes list when we clear deadnode list on refetchLocations. 
ignoredNodes list is only used on hedged read codepath

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
 2023-03-01 19:47:04 +00:00
Viraj Jasani 2ab7eb4caa
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432) 
Contributed by Viraj Jasani
 2023-03-01 18:53:10 +00:00
Viraj Jasani 28d2753d2f
HADOOP-18645. Provide keytab file key name with ServiceStateException (#5433) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-03-01 09:34:12 +08:00
rdingankar 0ca5686034
HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397) 
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
 2023-02-27 18:26:32 +00:00
Simbarashe Dzinamarira 61f369c43e
HDFS-16890: RBF: Ensures router periodically refreshes its record of a namespace's state. (#5298) 2023-02-27 17:56:24 +00:00
Viraj Jasani a90238c0b8
HADOOP-18631. Migrate Async appenders to log4j properties (#5418) 2023-02-26 01:47:44 +08:00
Owen O'Malley 8025a60ae7
HDFS-16901: Minor fix for unit test. 
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 16:37:49 -08:00
Simbarashe Dzinamarira 4cc33e5e37
HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-22 21:58:44 +00:00
hchaverr fb31393b65
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 10:38:50 -08:00
Viraj Jasani 88914cada0
HDFS-16925. Namenode audit log to only include IP address of client (#5407) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-21 08:26:32 +08:00
zhtttylz a3b500d046
HDFS-16916. Improve the use of JUnit Test in DFSClient (#5404) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:17:37 +08:00
hfutatzhanghb 723535b788
HDFS-16914. Add some logs for updateBlockForPipeline RPC. (#5381) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-16 09:25:50 +08:00
Zita Dombi 4cbe19f3a2
HDFS-16761. Namenode UI for Datanodes page not loading if any data node is down (#5390) 2023-02-15 16:16:04 +00:00
GuoPhilipse fe0541b58d
HDFS-16913. Fix flaky some unit tests since they offen timeout (#5377) 
Co-authored-by: gf13871 <gf13871@ly.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-14 15:29:38 +08:00
Viraj Jasani 90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Steve Vaughan f42c89dffb
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:31:32 +00:00
hfutatzhanghb f3c4277576
HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath (#5276) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-11 08:00:43 +08:00
Viraj Jasani 17c8cdf63c
HDFS-16907. ADDENDUM: Remove unused variables from testDataNodeMXBeanLastHeartbeats. (#5373) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-10 00:33:20 +05:30
huhaiyang 113a9e40cb
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301) 
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
 2023-02-09 13:28:34 +00:00
huhaiyang d5c046518e
HDFS-16910. Fix incorrectly initializing RandomAccessFile caused flush performance decreased for JN (#5359) 2023-02-09 10:47:57 +08:00
Steve Vaughan 08f58ecf07
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-08 17:15:42 +00:00
He Xiaoqiao 3ba058a894
HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 11:19:07 +08:00
He Xiaoqiao 7e919212c4
Revert "HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB." 
This reverts commit eb04ecd29d.
 2023-02-08 11:15:22 +08:00
hfutatzhanghb eb04ecd29d
HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 10:33:28 +08:00
hchaverri d310642626
HDFS-16895. [RBF] NamenodeHeartbeatService should use credentials of logged in user 2023-02-07 18:45:05 +00:00
Viraj Jasani f02c452cf1
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-07 09:54:40 +08:00
sunhao 0ae075a2af
HDFS-16848. RBF: Improve StateStoreZooKeeperImpl performance (#5147) 2023-02-05 09:33:05 +08:00
Viraj Jasani bce388fd3f
HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334) 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-03 09:11:02 +08:00
Masatake Iwasaki a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
huhaiyang 88c8ac750d
HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-01-31 17:46:38 +09:00
Szilard Nemeth b677d40ab5 HADOOP-18602. Remove netty3 dependency 2023-01-27 16:32:50 +01:00
zhtttylz 72b760130a
HDFS-16893. Standardize the usage of DFSClient debug log (#5303) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-01-19 07:56:41 +08:00
Viraj Jasani 04f3573f6a
HDFS-16891 Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel (#5300) 
Reviewed-by: Stephen O'Donnell <sodonnell@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-18 13:13:41 -08:00
Nikita Eshkeev 4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
Viraj Jasani 1263e024b9
HDFS-16887 Log start and end of phase/step in startup progress (#5292) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 14:26:52 -08:00
Simbarashe Dzinamarira f26d8bc9bd
HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 19:46:51 +00:00
huhaiyang e3b09b7512
HDFS-16884. Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed (#5280) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-01-11 09:12:26 +08:00
Chengbing Liu 4cf304de45
HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246) 
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-01-10 10:03:25 -08:00
huhaiyang f3cff032e6
HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed (#5283) 2023-01-10 09:02:56 +08:00
Simbarashe Dzinamarira b56d483258
HDFS-16876: Changes cleanup of shared RouterStateIdContext to be driven by namenodeResolver data. (#5282) 2023-01-09 23:55:13 +00:00
Simbarashe Dzinamarira cd19da1309
HDFS-16877: Enables state context for namenode in TestObserverWithRouter (#5257) 2023-01-07 00:18:35 +00:00
Tsz-Wo Nicholas Sze 5022003e0f
HDFS-16881. Warn if AccessControlEnforcer runs for a long time to check permission. (#5268) 2023-01-05 09:31:52 +08:00