Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,793 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24793 Commits

Author SHA1 Message Date
Steve Loughran 8814ba7849
HADOOP-18198. Add release 3.3.2 diff files 
Adds the 3.3.2 jdiff files which were left out of the
release commit...the ones the release doc didn't mention.

This is needed for the 3.3.3 build to complete.

Change-Id: I296103b7a51ac851ecc0503672af234519a7f0f9
 2022-04-20 16:50:35 +01:00
Steve Loughran 09badd76f1
HADOOP-17650. Bump solr to unblock build failure with Maven 3.8.1 (#2939) 
Reviewed-by: Siyao Meng <siyao@apache.org>

Contributed by Viraj Jasani

Change-Id: I67a6a1b7464044318a089bd82ffa51d9d37aea7f
 2022-04-20 16:42:48 +01:00
Dongjoon Hyun 369e26204c
HADOOP-17341. Upgrade commons-codec to 1.15 (#2428) 
Change-Id: Iab26db901570b507ab25ddbf316a9579a9e92620
Reviewed-by: Chao Sun <sunchao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-20 12:30:18 +01:00
Akira Ajisaka 1a1b108dbd
HADOOP-18178. Upgrade jackson to 2.13.2 and jackson-databind to 2.13.2.2 (#4147) 
(cherry picked from commit 4b786c797a)

 Conflicts:
	LICENSE-binary

Change-Id: I44042ade3c81a5852f1b81892ec9fbc18921a272
Co-authored-by: PJ Fanning <pjfanning@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-20 12:18:10 +01:00
Steve Loughran 2d0686e021
HADOOP-18202. create-release fails fatal: unsafe repository (#4188) 
Since April 2022/CVE-2022-24765, git refuses to work in directories
whose owner != the current user, unless explicitly told to trust it.

This patches the create-release script to trust the /build/source
dir mounted from the hosting OS, whose userid is inevitably different
from that of the account in the container running git.

Contributed by: Steve Loughran, Ayush Saxena and the new git error messages

Change-Id: I855a105e6d0ab533468f9436578c8d4f81b0840b
 2022-04-18 19:30:11 +01:00
André Fonseca 5212fde42f
HADOOP-18160 Avoid shading wildfly.openssl runtime dependency (#4074) 
Contributed by André Fonseca.

Change-Id: I198c66b58c0e05c1752ad9d6a50fe2f2932f61d5
 2022-04-18 15:08:19 +01:00
Masatake Iwasaki 877ef944f9
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-14 17:19:18 +01:00
litao d5845474e2
HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-04-14 17:19:18 +01:00
Kengo Seki 73c459db0c
MAPREDUCE-7373. Building MapReduce NativeTask fails on Fedora 34+ (#4120) 
(cherry picked from commit dc4a680da8)

Change-Id: Ia9ad34b5c3c0f767169fc48a1866c04ff73b1093
 2022-04-14 17:19:18 +01:00
GuoPhilipse 63c07519de
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)

Change-Id: I2cae5d1c27a492d896da5338a92c7a86f88a8b43
 2022-04-14 17:19:18 +01:00
Peter Bacsko 52aba525c3
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 
(cherry picked from commit a0deda1a77)

Change-Id: I935725ba094d2c35fdc91dd42883bf5b0d506d56
 2022-04-14 17:19:18 +01:00
qinyuren 376904e422
HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-04-14 17:19:18 +01:00
Wei-Chiu Chuang fd96d5c2d5
HADOOP-18155. Refactor tests in TestFileUtil (#4063) 
(cherry picked from commit d0fa9b5775)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Change-Id: I2bba28c56dd08da315856066b58b1778b67bfb45
Co-authored-by: Gautham B A <gautham.bangalore@gmail.com>
 2022-04-14 17:19:17 +01:00
Thinker313 5c3fdf0f4b
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-04-14 17:19:17 +01:00
Szilard Nemeth cb40b7d741
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 
Change-Id: I5505e1b8aaa394dfac31dade7aed6013e0279adc
 2022-04-14 17:19:17 +01:00
Szilard Nemeth a981df3aec
YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-04-14 17:19:17 +01:00
Ayush Saxena 51b3a5b22c
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)

Change-Id: I9f04082d650628bc1b8b62dacaaf472f8a578742
 2022-04-14 17:19:17 +01:00
Viraj Jasani c6b9fcfd6c
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation (#3991) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 697e5d4636)
 2022-04-14 17:19:17 +01:00
Chentao Yu 38d448e40b
HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-04-14 17:19:16 +01:00
daimin 686a934a5e
HDFS-16422. Fix thread safety of EC decoding during concurrent preads (#3881) 
(cherry picked from commit 0e74f1e467)

Change-Id: If28915934ed6f4ad7a68d280cadc8c563e2daaba
 2022-04-14 17:19:16 +01:00
singer-bin 980fab9168
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)

Change-Id: I03e4f2af17f0e4a8245c9c2c8ea1cb2cb41f777a
 2022-04-14 17:19:16 +01:00
Steve Loughran a1c0673526
HADOOP-18198. Preparing for 3.3.3 release 
Change-Id: Idebf79191dc91dad52073f2c63ee9ab3a99464d9
 2022-04-14 17:19:16 +01:00
Chao Sun f91452b289 Make upstream aware of 3.3.2 release 2022-03-02 19:09:35 -08:00
Steve Loughran 7f33a4e992
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I3856afa821dbc8c2e3cb1cbe33793ec1734e2e24
 2022-02-21 18:01:53 +00:00
Akira Ajisaka 0bcb014209 YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp (#2591) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9cb535caf2)
 2022-02-10 15:14:26 -08:00
Ayush Saxena f9cccfa7ea
HADOOP-18096. Distcp: Sync moves filtered file to home directory rather than deleting. (#3940). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: stack <stack@apache.org>
 2022-02-11 02:10:02 +05:30
Petre Bogdan Stolojan 8cd8e435fb
HADOOP-17198. Support S3 Access Points (#3260) (branch-3.3.2) (#3955) 
Add support for S3 Access Points. This provides extra security as it
ensures applications are not working with buckets belong to third parties.

To bind a bucket to an access point, set the access point (ap) ARN,
which must be done for each specific bucket, using the pattern

fs.s3a.bucket.$BUCKET.accesspoint.arn = ARN

* The global/bucket option `fs.s3a.accesspoint.required` to
mandate that buckets must declare their access point.
* This is not compatible with S3Guard.

Consult the documentation for further details.
 2022-02-04 10:09:00 -08:00
Steve Loughran 35e352a88b
HADOOP-18094. Disable S3A auditing by default. 
See HADOOP-18091. S3A auditing leaks memory through ThreadLocal references

* Adds a new option fs.s3a.audit.enabled to controls whether or not auditing
is enabled. This is false by default.

* When false, the S3A auditing manager is NoopAuditManagerS3A,
which was formerly only used for unit tests and
during filsystem initialization.

* When true, ActiveAuditManagerS3A is used for managing auditing,
allowing auditing events to be reported.

* updates documentation and tests.

This patch does not fix the underlying leak. When auditing is enabled,
long-lived threads will retain references to the audit managers
of S3A filesystem instances which have already been closed.

Contributed by Steve Loughran.

Change-Id: I671e594cd59e8ca77a1f65be791ad0ae9530b8d9
 2022-01-24 15:06:01 +00:00
Xiangyi Zhu 6da346a358 HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 09:26:05 -08:00
ahmarsuhail 78d57c8e58 HADOOP-16223. Remove misleading fs.s3a.delegation.tokens.enabled prompt (#3879) 
Contributed by Ahmar Suhail

Change-Id: I6a33043831a059325c58b0f76c925e52c6ae14f7
 2022-01-15 09:25:45 -08:00
Mukund Thakur 472a28f91d HADOOP-18065. ExecutorHelper.logThrowableFromAfterExecute() is too noisy. (#3860) 
Downgrading warn logs to debug in case of InterruptedException

Contributed By: Mukund Thakur
 2022-01-15 09:25:38 -08:00
monthonk d101e4d7fa HADOOP-14334. S3 SSEC tests to downgrade when running against a mandatory encryption object store (#3870) 
Contributed by Monthon Klongklaew

Change-Id: Ib275c9690bbc90170c6a442ded198fe006c20bc1
 2022-01-15 09:25:32 -08:00
Ayush Saxena 46b1411189 HADOOP-18056. DistCp: Filter duplicates in the source paths. (#3825). Contributed by Ayush Saxena. 
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2022-01-15 09:25:25 -08:00
Jackson Wang d7cacea07b HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 09:07:32 -08:00
Brahma Reddy Battula 8af28b7cca Updated the index as per 3.3.2 release 2022-01-05 10:44:28 -08:00
Ashutosh Gupta 6ea9906a8b HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:01:38 -08:00
liever18 d167ef4850 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 10:01:20 -08:00
Wei-Chiu Chuang e42e483d00 Make upstream aware of 3.3.1 release 2022-01-04 14:51:49 -08:00
Ashutosh Gupta b9ed31f9ec HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 14:38:52 -08:00
jianghuazhu b8fff25645 HADOOP-18063. Remove unused import AbstractJavaKeyStoreProvider in Shell class. (#3846) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7398a0f1b2)
 2022-01-04 14:38:41 -08:00
Igor Dvorzhak 9a24995d73 HADOOP-13464. Upgrade Gson dependency to version 2.8.9 (#2524) 
Change-Id: Ifd3fb9ec6ebfc8874bb799bc198219511fe55a2f

Update pom.xml

Update pom.xml

(cherry picked from commit 795054882a)
 2022-01-04 14:38:38 -08:00
Ashutosh Gupta 80a19c8a48 HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec88)
 2022-01-04 14:38:35 -08:00
Akira Ajisaka 6db1165380 Revert "HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836)" 
This reverts commit 05b43f2057.
 2022-01-04 14:38:31 -08:00
Ashutosh Gupta 8aac1c7dc9 HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec88)
 2022-01-04 14:38:28 -08:00
Akira Ajisaka 84a767ea24 HADOOP-18045. Disable TestDynamometerInfra (#3829) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
(cherry picked from commit dba139cd0f)
 2022-01-04 14:38:24 -08:00
jianghuazhu 2ddc66fc0c HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2022-01-04 14:38:19 -08:00
Akira Ajisaka 5a7efbfbb5 HADOOP-18052. Support Apple Silicon in start-build-env.sh (#3817) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c8725de547)
 2022-01-04 14:38:07 -08:00
Ashutosh Gupta c63509127f YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd)
 2022-01-04 14:38:02 -08:00
Eric Payne 1be9d60f6c YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 
(cherry picked from commit e2d6fd075d)
 2022-01-04 14:37:59 -08:00
wangzhaohui bceab2e439 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2022-01-04 14:37:55 -08:00