Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,059 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26059 Commits

Author SHA1 Message Date
Steve Loughran 906ae5138e
HADOOP-18402. S3A committer NPE in spark job abort (#4735) 
JobID.toString() and TaskID.toString() to only be called
when the IDs are not null.

This doesn't surface in MapReduce, but Spark SQL can trigger
in job abort, where it may invok abortJob() with an
incomplete TaskContext.

This patch MUST be applied to branches containing
HADOOP-17833. "Improve Magic Committer Performance."

Contributed by Steve Loughran.
 2022-08-15 11:20:59 +01:00
Steve Loughran eee59a8372
Revert "HADOOP-18402. S3A committer NPE in spark job abort (#4735)" 
(managed to commit through the github ui before I'd got the message done)

This reverts commit ad83e95046.
 2022-08-15 11:20:36 +01:00
Steve Loughran ad83e95046
HADOOP-18402. S3A committer NPE in spark job abort (#4735) 
jobId.toString() to only be called when the ID isn't null.

this doesn't surface in MR, but spark seems to manage it

Change-Id: I06692ef30a4af510c660d7222292932a8d4b5147
 2022-08-15 11:18:47 +01:00
slfan1989 ab88e4b65d
YARN-11223. [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router. (#4733) 2022-08-14 19:22:16 -07:00
Paul King d0fdb1d6e0
HADOOP-18404. Fix broken link to wiki help page in org.apache.hadoop.util.Shell (#4718). Contributed by Paul King. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-14 19:28:22 +05:30
slfan1989 d383cc4525
YARN-11236. Implement FederationReservationHomeSubClusterStore With MemoryStore. (#4711) 2022-08-13 10:37:20 -07:00
zhengchenyu 9f6bbc90a8
YARN-11148. In federation and security mode, nm recover may fail. (#4308) 2022-08-13 10:33:16 -07:00
kevins-29 b737869e01
HADOOP-18383. Codecs with @DoNotPool annotation are not closed causing memory leak (#4585) 2022-08-12 16:05:13 -07:00
xuzq e0c8c6eed4
HDFS-16678. RBF should supports disable getNodeUsage() in RBFMetrics (#4606) 2022-08-12 12:01:58 -07:00
xuzq 521e65acfe
HDFS-16723. Replace incorrect SafeModeException with StandbyException in RouterRpcServer.class (#4716). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-13 00:21:11 +05:30
Viraj Jasani 8c9533a0f8
HADOOP-18397. Shutdown AWSSecurityTokenService when its resources are no longer in use (#4722) 
Contributed by Viraj Jasani.
 2022-08-12 11:59:15 +01:00
xuzq 59619ad247
HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628) 2022-08-12 12:19:28 +08:00
Steve Vaughan 2005582a28
HDFS-16702. MiniDFSCluster should report cause of exception in assert error (#4680) 
When the MiniDFSClsuter detects that an exception caused an exit, it should include that exception as the cause for the AssertionError that it throws. The current AssertError simply reports the message "Test resulted in an unexpected exit" and provides a stack trace to the location of the check for an exit exception. This patch adds the original exception as the cause of the AssertError.
 2022-08-11 13:52:39 -07:00
slfan1989 6ca2d3f848
YARN-6539. Create SecureLogin inside Router. (#4712) 2022-08-11 13:25:51 -07:00
xuzq 09cabaad68
HDFS-13274. RBF: Extend RouterRpcClient to use multiple sockets (#4531) 2022-08-11 13:23:32 -07:00
Mukund Thakur b28e4c6904
HADOOP-18392. Propagate vectored s3a input stream stats to file system stats. (#4704) 
part of HADOOP-18103.

Contributed By: Mukund Thakur
 2022-08-12 01:42:00 +05:30
huaxiangsun e9509ac467
HADOOP-18340. deleteOnExit does not work with S3AFileSystem (#4608) 
Contributed by Huaxiang Sun
 2022-08-11 20:25:13 +01:00
Yubi Lee c0bbdca97e
HADOOP-18398. Prevent AvroRecord*.class from being included non-test jar (#4727) 
Contributed by Yubi Lee.
 2022-08-11 20:12:41 +01:00
slfan1989 133e8aabf0
YARN-11227. [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router. (#4715) 2022-08-10 14:53:46 -07:00
slfan1989 ffa9ed93a4
YARN-6572. Refactoring Router services to use common util classes for pipeline creations. (#4594) 2022-08-09 14:44:29 -07:00
Ashutosh Gupta 92abd99450
YARN-11237. Fix Bug while disabling proxy failover with Federation (#4658) 2022-08-08 13:29:27 -07:00
slfan1989 977f4b6165
MAPREDUCE-7385. impove JobEndNotifier#httpNotification With recommended methods. (#4403). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:59:03 +05:30
xuzq 895f7c51fd
HDFS-16709. Remove redundant cast in FSEditLogOp.class (#4667). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:54:01 +05:30
Viraj Jasani 06f0f7db79
HADOOP-18373. IOStatisticsContext tuning (#4705) 
The name of the option to enable/disable thread level statistics is
"fs.iostatistics.thread.level.enabled";

There is also an enabled() probe in IOStatisticsContext which can
be used to see if the thread level statistics is active.

Contributed by Viraj Jasani
 2022-08-08 10:42:57 +01:00
slfan1989 d8d3325d2f
HADOOP-18387. Fix incorrect placeholder in hadoop-common (#4679). Contributed by fanshilun. 2022-08-08 02:35:39 +05:30
Ashutosh Gupta 1cda2dcb6e
YARN-10793. Upgrade Junit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice (#4603) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-07 16:15:47 +09:00
slfan1989 52c2d99889
YARN-11228. [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router. (#4695) 2022-08-06 09:36:26 -07:00
xuzq 25ccdc77af
HDFS-16648. Add isDebugEnabled check for debug blockLogs in some classes (#4529) 2022-08-06 21:34:01 +08:00
Ashutosh Gupta bd0f9a46e1
HADOOP-18390. Fix out of sync import for HADOOP-18321 (#4694) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-06 21:51:23 +09:00
ahmarsuhail b5642c5638
HADOOP-18366. ITestS3Select.testSelectSeekFullLandsat is timing out. (#4702) 
Reduces size of data read to 1 MB

Contributed by Ahmar Suhail
 2022-08-05 14:13:04 +01:00
Steve Loughran 62dbefd8f2
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.
 2022-08-05 14:06:22 +01:00
Ayush Saxena 080e67039d
HADOOP-17234. Addendum. Add .asf.yaml to allow github and jira integration. (#4686). Contributed by Ayush Saxena. 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
 2022-08-05 08:34:56 +05:30
slfan1989 6f7c4c74ea
YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster (#4656) 2022-08-04 10:16:08 -07:00
Ashutosh Gupta 0aa08ef543
HADOOP-18363. Fix bug preventing hadoop-metrics2 from emitting metrics to > 1 Ganglia servers (#4627) 
* HADOOP-18363. Fix bug preventing hadoop-metrics2 from emitting metrics to > 1 Ganglia servers
 2022-08-04 18:26:38 +05:30
zhangshuyan0 dbf73e16b1
HADOOP-18364. All method metrics related to the RPC protocol should be initialized. (#4624). Contributed by Shuyan Zhang. 
Reviewed-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-04 13:00:37 +08:00
xuzq 8eebf40b1a
HDFS-16642. Moving the selecting inputstream from journalnode in EditLogTailer outof FSNLock (#4497) 2022-08-04 11:04:28 +08:00
Mukund Thakur 66dec9d322
HADOOP-18355. Update previous index properly while validating overlapping ranges. (#4647) 
part of HADOOP-18103.

Contributed By: Mukund Thakur
 2022-08-04 04:08:04 +05:30
slfan1989 c5ec727435
YARN-11230. [Federation] Add getContainer, signalToContainer REST APIs for Router. (#4689) 2022-08-03 11:21:48 -07:00
slfan1989 6463f86f83
YARN-11029. Refactor AMRMProxy Service code and Added Some Metrics. (#4650) 2022-08-03 09:38:00 -07:00
slfan1989 c5eba323bc
YARN-6972. Adding RM ClusterId in AppInfo. (#4673) 2022-08-03 09:35:40 -07:00
Steve Vaughan 0fc7dd8228
HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) 2022-08-03 09:14:11 -07:00
xuzq 0f36539d60
HDFS-16712. Fix incorrect placeholder in DataNode.java (#4672). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-03 13:01:41 +05:30
Ashutosh Gupta 69f6fdb757
HADOOP-18301. Upgrade commons-io to 2.11.0 (#4455) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-03 10:44:39 +09:00
slfan1989 1f0a71a92b
YARN-8973. [Router] Add missing methods in RMWebProtocol. (#4664) 2022-08-02 14:07:09 -07:00
slfan1989 57da4bb0a1
YARN-11220. [Federation] Add getLabelsToNodes, getClusterNodeLabels, getLabelsOnNode REST APIs for Router (#4657) 2022-08-02 12:09:55 -07:00
ahmarsuhail 123d1aa884
HADOOP-18368. Fixes ITestCustomSigner for access point names with '-' (#4634) 
Contributed By: Ahmar Suhail <ahmarsu@amazon.co.uk>
 2022-08-02 01:49:42 +05:30
slfan1989 13fbfd5dea
HADOOP-18358. Update commons-math3 from 3.1.1 to 3.6.1. (#4619). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-02 01:48:47 +05:30
slfan1989 2680f17eb4
YARN-11180. Refactor some code of getNewApplication, submitApplication etc. (#4618) 2022-07-29 08:23:11 -07:00
slfan1989 e994635a95
YARN-11212. [Federation] Add getNodeToLabels REST APIs for Router. (#4614) 2022-07-28 11:53:04 -07:00
Mukund Thakur a5b12c8010
HADOOP-18227. Add input stream IOStats for vectored IO api in S3A. (#4636) 
part of HADOOP-18103.

Contributed By: Mukund Thakur
 2022-07-28 21:57:37 +05:30