Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,731 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

25731 Commits

Author SHA1 Message Date
Anmol Asrani 7c97c0f969
HADOOP-18084. ABFS: Add testfilePath while verifying test contents are read correctly (#3903) 
Contributed by: Anmol Asrani
 2022-01-19 10:13:13 +00:00
qinyuren db2c3200e6
HDFS-16423. Balancer should not get blocks on stale storages (#3883) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-19 17:59:42 +09:00
litao e355646330
HDFS-16399. Reconfig cache report parameters for datanode (#3841) 2022-01-19 15:10:39 +09:00
qinyuren fcb1076699
HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 2022-01-19 13:42:33 +09:00
Steve Loughran d8ab84275e
HADOOP-18068. upgrade AWS SDK to 1.12.132 (#3864) 
With this update, the versions of key shaded dependencies are

  jackson    2.12.3
  httpclient 4.5.13

Contributed by Steve Loughran
 2022-01-18 10:31:28 +00:00
Steve Loughran 14ba19af06
HADOOP-17409. Remove s3guard from S3A module (#3534) 
Completely removes S3Guard support from the S3A codebase.

If the connector is configured to use any metastore other than
the null and local stores (i.e. DynamoDB is selected) the s3a client
will raise an exception and refuse to initialize.

This is to ensure that there is no mix of S3Guard enabled and disabled
deployments with the same configuration but different hadoop releases
-it must be turned off completely.

The "hadoop s3guard" command has been retained -but the supported
subcommands have been reduced to those which are not purely S3Guard
related: "bucket-info" and "uploads".

This is major change in terms of the number of files
changed; before cherry picking subsequent s3a patches into
older releases, this patch will probably need backporting
first.

Goodbye S3Guard, your work is done. Time to die.

Contributed by Steve Loughran.
 2022-01-17 18:08:57 +00:00
Gera Shegalov a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-17 16:20:48 +09:00
Jackson Wang d8862822d2
HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 22:38:11 +09:00
litao f02374df92
HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 14:48:10 +09:00
Xiangyi Zhu 034dc8d03c
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3063). Contributed by Xiangyi Zhu. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-13 12:56:19 +08:00
ahmarsuhail 7542677470
HADOOP-16223. Remove misleading fs.s3a.delegation.tokens.enabled prompt (#3879) 
Contributed by Ahmar Suhail
 2022-01-12 17:25:17 +00:00
Gautham B A 271ac0e091
HDFS-16419. Make HDFS data transfer tools cross platform (#3873) 2022-01-12 19:57:27 +05:30
Viraj Jasani 93294f0329
HADOOP-18077. ProfileOutputServlet unable to proceed due to NPE (#3875) 2022-01-12 16:20:34 +08:00
luoyuan3471 e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
 2022-01-12 11:40:32 +08:00
Andras Gyori 21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable (#3859) 2022-01-11 14:38:59 +08:00
Felix Nguyen b88d5ae1c6
HDFS-16417. StaticRouterRpcFairnessPolicyController init (#3871) 2022-01-10 18:28:20 +08:00
monthonk b27732c69b
HADOOP-14334. S3 SSEC tests to downgrade when running against a mandatory encryption object store (#3870) 
Contributed by Monthon Klongklaew
 2022-01-09 18:01:47 +00:00
litao 39efbc6b6f
HDFS-16404. Fix typo for CachingGetSpaceUsed (#3844). Contributed by tomscut. 2022-01-09 16:41:10 +08:00
Viraj Jasani 08c803ea30
MAPREDUCE-7371. DistributedCache alternative APIs should not use DistributedCache APIs internally (#3855) 2022-01-09 00:18:10 +09:00
Viraj Jasani f64fda0f00
HADOOP-18055. Async Profiler endpoint for Hadoop daemons (#3824) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-06 17:56:49 +08:00
Mukund Thakur da0a6ba1ce
HADOOP-18065. ExecutorHelper.logThrowableFromAfterExecute() is too noisy. (#3860) 
Downgrading warn logs to debug in case of InterruptedException

Contributed By: Mukund Thakur
 2022-01-06 10:54:27 +05:30
litao 46b02788bd
HDFS-16371. Exclude slow disks when choosing volume (#3753) 2022-01-06 09:31:00 +09:00
Ayush Saxena 657a2882e9
HADOOP-18056. DistCp: Filter duplicates in the source paths. (#3825). Contributed by Ayush Saxena. 
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2022-01-05 23:53:07 +05:30
Ashutosh Gupta 43e5218a86
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 15:51:14 +00:00
liever18 e1d0aa9ee5
HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 2022-01-05 15:42:58 +00:00
Gautham B A c3006be516
HDFS-16407. Make hdfs_du tool cross platform (#3848) 2022-01-04 22:29:54 +05:30
Ashutosh Gupta 9eea0e28f2
HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 13:25:16 +09:00
jianghuazhu 7398a0f1b2
HADOOP-18063. Remove unused import AbstractJavaKeyStoreProvider in Shell class. (#3846) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 11:25:13 +09:00
Ayush Saxena 61d424f385
HDFS-16393. RBF: Fix TestRouterRPCMultipleDestinationMountTableResolver. (#3849). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-01-04 07:50:01 +05:30
Ayush Saxena 0c51dec124
HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 07:48:03 +05:30
minni31 3bf014d871
YARN-8859. Add audit logs for router service (#3607) 
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-01-03 10:38:59 +05:30
jianghuazhu 43afd1753a
HDFS-16394.RPCMetrics increases the number of handlers in processing. (#3822) 2021-12-31 16:40:14 +08:00
Igor Dvorzhak 795054882a
HADOOP-13464. Upgrade Gson dependency to version 2.8.9 (#2524) 
Change-Id: Ifd3fb9ec6ebfc8874bb799bc198219511fe55a2f

Update pom.xml

Update pom.xml
 2021-12-30 13:27:54 -08:00
Symious b225287913
HDFS-16348. Mark slownode as badnode to recover pipeline (#3704) 2021-12-30 00:27:11 +09:00
Ashutosh Gupta caab29ec88
HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-28 21:44:38 +09:00
Akira Ajisaka dba139cd0f
HADOOP-18045. Disable TestDynamometerInfra (#3829) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
 2021-12-28 13:22:12 +09:00
Ashutosh Gupta ebdbe7eb82
HADOOP-18057. Fix typo: validateEncrytionSecrets -> validateEncryptionSecrets (#3826) 2021-12-27 16:51:17 +08:00
jianghuazhu a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 17:24:12 +09:00
Cheng Pan 4db27acf06
YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 11:08:04 +09:00
KevinWikant d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning (#3675) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 22:59:08 +09:00
Akira Ajisaka c8725de547
HADOOP-18052. Support Apple Silicon in start-build-env.sh (#3817) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-12-23 18:13:18 +09:00
Ashutosh Gupta 00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 17:14:51 +09:00
Giovambattista Vieri 97ed0293ee
Update BUILDING.txt (#3811) 
java-8-openjdk become openjdk-8-jdk (see both ubuntu and debian package's name)

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-22 13:08:14 +08:00
secfree cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout (#3821) 2021-12-22 10:44:07 +08:00
Eric Payne e2d6fd075d YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:05:39 +00:00
wangzhaohui 8d251bd629
HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 2021-12-21 11:28:57 +00:00
Jackson Wang 9420cb6f8a
HDFS-16385. Fix Datanode retrieve slownode information bug. (#3803) 
Reviewed-by: Symious <yiyang0203@gmail.com>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
 2021-12-21 11:04:02 +09:00
jianghuazhu 746b328554
HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 2021-12-20 11:28:55 +00:00
secfree 32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout (#3815) 2021-12-20 18:16:22 +08:00
Viraj Jasani 6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled (#3802) 2021-12-20 18:02:57 +08:00