Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,632 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

2998 Commits

Author SHA1 Message Date
Steve Loughran e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-11 17:57:25 +01:00
slfan1989 a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. (#5095). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 01:42:12 +05:30
Tak Lon (Stephen) Wu 0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) 
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu
 2023-05-03 11:05:55 +01:00
Steve Loughran b6b9bd67bb
MAPREDUCE-7437. MR Fetcher class to use an AtomicInteger to generate IDs. (#5579) 
...as until now it wasn't thread safe

Contributed by Steve Loughran
 2023-04-25 19:53:40 +01:00
Sebastian Baunsgaard 6aac6cb212
HADOOP-18660. Filesystem Spelling Mistake (#5475). Contributed by Sebastian Baunsgaard. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 21:44:04 +05:30
Steve Loughran 405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
zhaixiaojuan@loongson.cn 028cde0006
HADOOP-18644. Add bswap support for LoongArch64. (#5453). Contributed by zhaixiaojuan. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 11:08:59 +08:00
Viraj Jasani cf4a678ce9
HADOOP-18649. CLA and CRLA appenders to be replaced with RFA (#5448) 2023-03-16 00:46:17 +08:00
Viraj Jasani 405bfa2800
HADOOP-18654. Remove unused custom appender TaskLogAppender (#5457) 2023-03-16 00:45:37 +08:00
Szilard Nemeth 8f6be3678d MAPREDUCE-7434. Fix ShuffleHandler tests. Contributed by Tamas Domok 2023-03-01 16:10:05 +01:00
Ayush Saxena fe5bb49ad9
Revert "YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure. Contributed by Susheel Gupta" 
This reverts commit 8eda456d37.
 2023-02-22 07:28:13 +05:30
Viraj Jasani 90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Tamas Domok e4b5314991
MAPREDUCE-7433. Remove unused mapred/LoggingHttpResponseEncoder.java. (#5388) 2023-02-13 16:21:27 +01:00
Tamas Domok 151b71d7af
MAPREDUCE-7431. ShuffleHandler refactor and fix after Netty4 upgrade. (#5311) 2023-02-10 17:40:21 +01:00
Szilard Nemeth 815cde9810 YARN-6971. Clean up different ways to create resources. Contributed by Riya Khandelwal 2023-01-25 17:28:29 +01:00
Szilard Nemeth 8eda456d37 YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure. Contributed by Susheel Gupta 2023-01-25 15:06:20 +01:00
Nikita Eshkeev 4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
Ashutosh Gupta 38453f8589
MAPREDUCE-7413. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-hs-plugins (#5023) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2023-01-16 18:19:28 +09:00
Ashutosh Gupta 082266516a
MAPREDUCE-7417. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-uploader (#5019) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2023-01-16 17:22:04 +09:00
skysiders 36bf54aba0
MAPREDUCE-7375 JobSubmissionFiles don't set right permission after mkdirs (#4237) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 13:48:29 -08:00
Akira Ajisaka 049d1762bd
MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils (#5243) 
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
 2022-12-24 04:33:35 +09:00
Ashutosh Gupta 85ec7969a7
MAPREDUCE-7428. Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp (#5209) 
Contributed by: Ashutosh Gupta
 2022-12-14 12:54:08 +00:00
Ashutosh Gupta dcde414570
MAPREDUCE-7422. Upgrade Junit 4 to 5 in hadoop-mapreduce-examples (#5029) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 14:36:35 +09:00
Szilard Nemeth 142df247ed YARN-11369. Commons.compress throws an IllegalArgumentException with large uids after 1.21. Contributed by Benjamin Teke 2022-11-16 13:07:05 +01:00
Ashutosh Gupta a48e8c9beb
MAPREDUCE-5608. Replace and deprecate mapred.tasktracker.indexcache.mb (#5014) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-14 11:07:40 +09:00
slfan1989 04b31d7ecf
MAPREDUCE-7390. Remove WhiteBox in mapreduce module. (#4462) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-14 10:45:20 +09:00
Szilard Nemeth 5bb11cecea HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4 #3259. Contributed by Szilard Nemeth. 2022-11-11 09:05:01 +01:00
Steve Vaughan 2ba982a061
MAPREDUCE-7386. Maven parallel builds (skipping tests) fail (#4415) 
Contributed by Steve Vaughan Jr
 2022-11-04 11:50:43 +00:00
Ashutosh Gupta e62ba16a02
HADOOP-18484. Upgrade hsqldb to v2.7.1 to mitigate CVE-2022-41853 (#4991) 2022-11-02 08:41:27 +01:00
wangteng13 388f2f182f
document fix for MAPREDUCE-7425 (#5090) 
Reviewed-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-11-01 13:34:59 -07:00
Samrat e04c9e810b
MAPREDUCE-7426. Fix typo in StartEndTimeBase (#4894) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 04:23:30 +09:00
PJ Fanning aac87ffe76
MAPREDUCE-7411: use secure XML parsers in mapreduce modules (#4980) 
Lockdown of parsers in hadoop-mapreduce.

Follow-on to HADOOP-18469. Add secure XML parser factories to XMLUtils

Contributed by P J Fanning
 2022-10-21 14:02:11 +01:00
Ashutosh Gupta 062c50db6b
MAPREDUCE-7370. Parallelize MultipleOutputs#close call (#4248). Contributed by Ashutosh Gupta. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-10-06 15:23:05 -07:00
Ashutosh Gupta 30c36ef25a
HADOOP-18400. Fix file split duplicating records from a succeeding split when reading BZip2 text files (#4732) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-19 13:45:05 +09:00
Ashutosh Gupta 59d3c20118
MAPREDUCE-7407. Avoid stopContainer() on dead node (#4779) 2022-09-15 10:30:36 -07:00
sreeb-msft c48ed3e96c
HADOOP-18408. ABFS: ITestAbfsManifestCommitProtocol fails on nonHNS configuration (#4758) 
ITestAbfsManifestCommitProtocol  to set requireRenameResilience to false for nonHNS configuration  (#4758)

Contributed by Sree Bhattacharyya
 2022-09-02 12:33:12 +01:00
9uapaw 84081a8cae MAPREDUCE-7409. Make shuffle key length configurable. Contributed by Ashutosh Gupta. 2022-08-31 17:32:51 +02:00
Steve Loughran de37fd37d6
MAPREDUCE-7403. manifest-committer dynamic partitioning support. (#4728) 
Declares its compatibility with Spark's dynamic
output partitioning by having the stream capability
"mapreduce.job.committer.dynamic.partitioning"

Requires a Spark release with SPARK-40034, which
does the probing before deciding whether to 
accept/rejecting instantiation with
dynamic partition overwrite set

This feature can be declared as supported by
any other PathOutputCommitter implementations
whose algorithm and destination filesystem
are compatible.

None of the S3A committers are compatible.

The classic FileOutputCommitter is, but it
does not declare itself as such out of our fear
of changing that code. The Spark-side code
will automatically infer compatibility if
the created committer is of that class or
a subclass.

Contributed by Steve Loughran.
 2022-08-24 11:18:19 +01:00
slfan1989 977f4b6165
MAPREDUCE-7385. impove JobEndNotifier#httpNotification With recommended methods. (#4403). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:59:03 +05:30
Ashutosh Gupta bd0f9a46e1
HADOOP-18390. Fix out of sync import for HADOOP-18321 (#4694) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-06 21:51:23 +09:00
Steve Loughran 62dbefd8f2
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.
 2022-08-05 14:06:22 +01:00
skysiders 9fe96238d2
MAPREDUCE-7372 MapReduce set permission too late in copyJar method (#4026). Contributed by Zhang Dongsheng. 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-07-25 11:38:59 -07:00
Masatake Iwasaki 3cce41a1f6 Make upstream aware of 3.2.4 release. 
(cherry picked from commit e1637a57df)
 2022-07-22 02:27:19 +00:00
PJ Fanning 34e548cb62
HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api (#4547) 
This downgrades jackson from the version switched to in
    HADOOP-18033 (2.13.0), to Jackson 2.12.7.
    This removes the dependency on javax.ws.rs-api,
    so avoiding runtime problems with applications using
    jersey-core v1 and/or jsr311-api.
    
    The 2.12.7 release still contains the fix for CVE-2020-36518.
    
    Contributed by PJ Fanning
 2022-07-17 21:37:54 +05:30
Ashutosh Gupta 4e8c0b902e
MAPREDUCE-7201.Make Job History File Permissions configurable (#4507) 
* MAPREDUCE-7201.Make Job History File Permissions configurable

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-07-11 11:34:52 +05:30
Akira Ajisaka 9b1d3579b4
Revert "MAPREDUCE-7388. Remove unused variable _eof in GzipCodec.cc (#4429)" 
This reverts commit fac895828f.
 2022-07-09 03:05:42 +09:00
cfg1234 fac895828f
MAPREDUCE-7388. Remove unused variable _eof in GzipCodec.cc (#4429) 
Co-authored-by: cWX456268 <chenfengge1@huawei.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-07-09 02:51:49 +09:00
Ashutosh Gupta a432925f74
HADOOP-18321.Fix when to read an additional record from a BZip2 text file split (#4521) 
* HADOOP-18321.Fix when to read an additional record from a BZip2 text file split

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com> and Reviewed by Akira Ajisaka.
 2022-07-06 10:00:14 +05:30
slfan1989 073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
Steve Loughran c9ddbd210c
MAPREDUCE-7391. TestLocalDistributedCacheManager failing after HADOOP-16202 (#4472) 
Fixing a mockito-based test which broke when HADOOP-16202
changed the methods being invoked.

Contributed by Steve Loughran
 2022-06-22 12:52:41 +01:00