Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
23,100 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

23100 Commits

Author SHA1 Message Date
avijayanhwx f3eaa84f9d HDDS-2164 : om.db.checkpoints is getting filling up fast. (#1536) 2019-10-04 12:44:21 -07:00
Norbert Kalmar 10bdc592d6 HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár. 2019-10-04 10:57:07 -07:00
Steve Loughran 6574f27fa3
HADOOP-16570. S3A committers encounter scale issues. 
Contributed by Steve Loughran.

This addresses two scale issues which has surfaced in large scale benchmarks
of the S3A Committers.

* Thread pools are not cleaned up.
  This now happens, with tests.

* OOM on job commit for jobs with many thousands of tasks,
  each generating tens of (very large) files.

Instead of loading all pending commits into memory as a single list, the list
of files to load is the sole list which is passed around; .pendingset files are
loaded and processed in isolation -and reloaded if necessary for any
abort/rollback operation.

The parallel commit/abort/revert operations now work at the .pendingset level,
rather than that of individual pending commit files. The existing parallelized
Tasks API is still used to commit those files, but with a null thread pool, so
as to serialize the operations.

Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797
 2019-10-04 18:54:22 +01:00
Eric Yang aa24add8f0 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle
 2019-10-04 13:13:10 -04:00
Elek, Márton 3f166512af HDDS-2237. KeyDeletingService throws NPE if it's started too early (#1584) 2019-10-04 09:48:49 -07:00
Sunil G 4510970e2f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 2019-10-04 21:49:07 +05:30
Doroszlai, Attila f826420d2b
HDDS-2230. Invalid entries in ozonesecure-mr config. (Addendum) 2019-10-04 17:37:51 +02:00
Tsz-Wo Nicholas Sze 531cc938fe
HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C. (#1595) 2019-10-04 21:16:28 +08:00
Steve Loughran f44abc3e11
HADOOP-16207 Improved S3A MR tests. 
Contributed by Steve Loughran.

Replaces the committer-specific terasort and MR test jobs with parameterization
of the (now single tests) and use of file:// over hdfs:// as the cluster FS.

The parameterization ensures that only one of the specific committer tests
run at a time -overloads of the test machines are less likely, and so the
suites can be pulled back into the parallel phase.

There's also more detailed validation of the stage outputs of the terasorting;
if one test fails the rest are all skipped. This and the fact that job
output is stored under target/yarn-${timestamp} means failures should
be more debuggable.

Change-Id: Iefa370ba73c6419496e6e69dd6673d00f37ff095
 2019-10-04 14:12:31 +01:00
cxorm bca014b0e0
HDDS-2216. Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in compose .env files. 
Closes #1570.
 2019-10-04 14:57:33 +02:00
S O'Donnell 6171a41b4c
HDDS-2199. In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host 
Closes #1551
 2019-10-04 14:00:06 +02:00
dchitlangia d061c8469f
HDDS-2140. Add robot test for GDPR feature 
Closes #1542
 2019-10-04 13:51:57 +02:00
Márton Elek bffcd33085
HDDS-2230. Invalid entries in ozonesecure-mr config 
Doroszlai, Attila <adoroszlai@apache.org>
 2019-10-04 13:24:42 +02:00
Tsz-Wo Nicholas Sze a9849f65ba
Revert "HDDS-2222 (#1578)" (#1594) 
This reverts commit 4cf0b3660f.
 2019-10-04 19:19:45 +08:00
Tsz-Wo Nicholas Sze 4cf0b3660f
HDDS-2222 (#1578) 
Thanks @jnp  for reviewing this.  Merging now.
 2019-10-04 17:50:21 +08:00
Abhishek Modi 2478cbafe6 YARN-9782. Avoid DNS resolution while running SLS. Contributed by Abhishek Modi. 2019-10-04 14:45:10 +05:30
Takanobu Asanuma b23bdaf085 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 2019-10-04 16:47:06 +09:00
Doroszlai, Attila ec8f691201
HDDS-2225. SCM fails to start in most unsecure environments due to leftover secure config 
Closes #1568
 2019-10-04 08:50:13 +02:00
Stephen O'Donnell c99a12167f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 22:13:50 -07:00
Virajith Jalaparti 844b766da5 HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. (#1573)" 2019-10-03 21:32:48 -07:00
Nanda kumar 9700e2003a
HDDS-2223. Support ReadWrite lock in LockManager. (#1564) 2019-10-04 08:32:43 +05:30
Nanda kumar cdaa480dbf
HDDS-2198. SCM should not consider containers in CLOSING state to come out of safemode. (#1540) 2019-10-04 08:20:58 +05:30
pingsutw 1dde3efb91
HADOOP-16624. Upgrade hugo to the latest version in Dockerfile 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-10-04 10:01:07 +09:00
Aravindan Vijayan b7cb8fe07c HDDS-2200 : Recon does not handle the NULL snapshot from OM DB cleanly. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-03 15:38:19 -07:00
Aravindan Vijayan 76605f17dd HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-03 15:17:09 -07:00
Doroszlai, Attila 944668674b HDDS-2231. test-single.sh cannot copy results (#1575) 2019-10-03 14:50:04 -07:00
Doroszlai, Attila 47d721d7dd HDDS-2234. rat.sh fails due to ozone-recon-web/build files (#1580) 2019-10-03 14:46:50 -07:00
Doroszlai, Attila 51eaecab20 HDDS-2211. Collect docker logs if env fails to start (#1553) 2019-10-03 12:44:04 -07:00
Ayush Saxena a3fe404244 HDFS-14881. Safemode 'forceExit' option, doesn’t shown in help message. Contributed by Renukaprasad C. 2019-10-03 22:34:55 +05:30
Anu Engineer d59bcbfa0f
HDDS-2226. S3 Secrets should use a strong RNG. (#1572) 2019-10-03 09:28:41 -07:00
Ayush Saxena 5a7483ca5c HDFS-14888. RBF: Enable Parallel Test Profile for builds. Contributed by Ayush Saxena. 2019-10-03 19:21:27 +05:30
Sneha Vijayarajan c19fa3d4fe HADOOP-16605. Fix testcase testSSLChannelModeConfig 
Contributed by Sneha Vijayarajan.
 2019-10-03 11:13:55 +01:00
Akira Ajisaka 0e026cb0ce
HADOOP-16620. [pb-upgrade] Remove protocol buffers 3.7.1 from requirements in BUILDING.txt (#1558) 
* Removed protobuf 3.7.1 from Dockerfile
 2019-10-03 15:24:47 +09:00
Xiaoyu Yao c5665b23ca
HDDS-2228. Fix NPE in OzoneDelegationTokenManager#addPersistedDelegat… (#1571) 2019-10-02 23:09:06 -07:00
Anu Engineer 4c24f2434d HDDS-2072. Make StorageContainerLocationProtocolService message based 
Contributed by Elek, Marton.
 2019-10-02 17:00:53 -07:00
Chen Liang 1303255aee HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:38:20 -07:00
Siddharth Seth 559ee277f5
HADOOP-16599. Allow a SignerInitializer to be specified along with a Custom Signer 2019-10-02 16:03:48 -07:00
Bharat Viswanadham b09d389001
HDDS-2019. Handle Set DtService of token in S3Gateway for OM HA. (#1489) 2019-10-02 15:41:54 -07:00
Bharat Viswanadham 53ed78bcdb
HDDS-2224. Fix loadup cache for cache cleanup policy NEVER. (#1567) 2019-10-02 15:18:43 -07:00
Bharat Viswanadham 169cef758d
HDDS-2162. Make OM Generic related configuration support HA style config. (#1511) 2019-10-02 15:09:33 -07:00
Anu Engineer 685918ef41
HDDS-2227. GDPR key generation could benefit from secureRandom. (#1574) 2019-10-02 12:34:53 -07:00
Anu Engineer ffd4e52725 HDDS-2073. Make SCMSecurityProtocol message based. 
Contributed by Elek, Marton.
 2019-10-02 12:19:58 -07:00
Márton Elek e8ae632d4c HDDS-2068. Make StorageContainerDatanodeProtocolService message based 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-02 11:18:41 -07:00
Sunil G 0d2d6f9267 YARN-9792. Document examples of SchedulerConf with Node Labels. Contributed by Prabhu Joseph 2019-10-02 23:43:47 +05:30
Anu Engineer 2e1fd44596
HDDS-2201. Rename VolumeList to UserVolumeInfo. (#1566) 2019-10-02 09:25:47 -07:00
Abhishek Modi 61a8436004 YARN-9870. Remove unused function from OpportunisticContainerAllocatorAMService. Contributed by Abhishek Modi. 2019-10-02 17:55:07 +05:30
Doroszlai, Attila f1ba9bfad7
HDDS-2187. ozone-mr test fails with No FileSystem for scheme "o3fs" 
Closes #1537
 2019-10-02 12:57:23 +02:00
bshashikant 41440ec890
HDDS-2210. ContainerStateMachine should not be marked unhealthy if applyTransaction fails with closed container exception(#1552). 2019-10-02 10:11:01 +05:30
Akira Ajisaka 3df733c250
HDFS-14885. UI: Fix a typo on WebUI of DataNode. Contributed by Xieming Li. 2019-10-02 11:03:16 +09:00
Eric Yang d1ddba60e5 YARN-9862. Increase yarn-services-core test timeout value. 
Contributed by Prabhu Joseph
 2019-10-01 21:15:26 -04:00