9e69628f55
HADOOP-16455. ABFS: Implement FileSystem.access() method.
...
Contributed by Bilahari T H.
2019-11-27 15:56:38 +00:00
ea25f4de23
HADOOP-16709. S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths ( #1721 ). Contributed by Gabor Bota.
2019-11-26 16:36:19 +01:00
de38045021
HADOOP-16687. ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account. ( #1701 )
2019-11-21 11:24:12 +09:00
b033c681e4
HADOOP-16612. Track Azure Blob File System client-perceived latency
...
Contributed by Jeetesh Mangwani.
This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.
The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.
2019-11-19 09:00:24 -08:00
9fbfe6c8f9
HADOOP-16632 Speculating & Partitioned S3A magic committers can leave pending files under __magic ( #1599 )
...
Contributed by Steve Loughran.
This downgrade the checks for leftover __magic entries from fail to warn now the parallel
test runs make speculation more likely.
Change-Id: Ia4df2e90f82a06dbae69f3fdaadcbb0e0d713b38
2019-11-19 13:54:33 +00:00
cad540819f
HADOOP-16484. S3A to warn or fail if S3Guard is disabled - addendum: silent for S3GuardTool ( #1714 ). Contributed by Gabor Bota.
...
Change-Id: I63b928ef5da425ef982dd4100a426fc23f64bac1
2019-11-18 13:56:37 +01:00
96c4520f89
HADOOP-16710. Testing_azure.md documentation is misleading.
...
Contributed by Andras Bokor.
Change-Id: Icf07a53145936953629c7dace2e9648b7b21588d
2019-11-17 17:04:29 +00:00
990063d2af
HADOOP-16665. Filesystems to be closed if they failed during initialize().
...
Contributed by Steve Loughran.
This FileSystem instantiation so if an IOException or RuntimeException is
raised in the invocation of FileSystem.initialize() then a best-effort
attempt is made to close the FS instance; exceptions raised that there
are swallowed.
The S3AFileSystem is also modified to do its own cleanup if an
IOException is raised during its initialize() process, it being the
FS we know has the "potential" to leak threads, especially in
extension points (e.g AWS Authenticators) which spawn threads.
Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04
2019-11-12 18:17:21 +00:00
f6697aa82b
HADOOP-16477. S3A delegation token tests fail if fs.s3a.encryption.key set.
...
Contributed by Steve Loughran.
Change-Id: I843989f32472bbdefbd4fa504b26c7a614ab1cee
2019-11-12 15:31:53 +00:00
d17ba85482
HADOOP-16681. mvn javadoc:javadoc fails in hadoop-aws. Contributed by Xieming Li
2019-11-05 15:24:59 +09:00
dca19fc3aa
HADOOP-16484. S3A to warn or fail if S3Guard is disabled ( #1661 ). Contributed by Gabor Bota.
2019-11-04 12:55:36 +01:00
477505ccfc
HDFS-14824. [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results. ( #1685 )
2019-11-01 09:32:32 -07:00
e32ab5e179
HDFS-14907. [Dynamometer] Add JUnit JAR to classpath for SimulatedDataNodes ( #1680 )
2019-10-29 08:21:16 -07:00
d5e9971e6d
HADOOP-16653. S3Guard DDB overreacts to no tag access ( #1660 ). Contributed by Gabor Bota.
2019-10-28 11:22:41 +01:00
b41394eec8
HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure. ( #1673 )
2019-10-24 16:25:31 -07:00
1d5d7d0989
HADOOP-16658. S3A connector does not support including the token renewer in the token identifier.
...
Contributed by Phil Zampino.
Change-Id: Iea9d5028dcf58bda4da985604f5cd3ac283619bd
2019-10-23 16:32:49 +01:00
3d41f33018
HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng.
...
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-18 08:25:02 -07:00
9a8edb0aed
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI
2019-10-16 13:14:15 -07:00
bbcf0b91d6
HADOOP-16478. S3Guard bucket-info fails if the caller lacks s3:GetBucketLocation.
...
Contributed by Steve Loughran.
Includes HADOOP-16651. S3 getBucketLocation() can return "US" for us-east.
Change-Id: Ifc0dca76e51495ed1a8fc0f077b86bf125deff40
2019-10-16 09:41:33 +01:00
fabd41fa48
HDFS-14856. Fetch file ACLs while mounting external store. ( #1478 )
2019-10-14 09:44:56 -07:00
74e5018d87
HADOOP-16635. S3A "directories only" scan still does a HEAD.
...
Contributed by Steve Loughran.
Change-Id: I5e41d7f721364c392e1f4344db83dfa8c5aa06ce
2019-10-14 17:05:52 +01:00
dee9e97075
Revert "HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos."
...
This reverts commit 7a4b3d42c4
2019-10-14 16:56:50 +01:00
62b5cefaea
YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal
2019-10-11 13:39:27 +02:00
4a700c20d5
HADOOP-16520. Race condition in DDB table init and waiting threads. ( #1576 ). Contributed by Gabor Bota.
...
Fixes HADOOP-16349. DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry
Change-Id: Ia83e92b9039ccb780090c99c41b4f71ef7539d35
2019-10-11 12:08:47 +02:00
7a4b3d42c4
HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos.
...
Contributed by lqjacklee.
Change-Id: I32bb00a683102e7ff8ff8ce0b8d9c3195ca7381c
2019-10-10 21:58:42 +01:00
effe6087a5
HADOOP-16650. ITestS3AClosedFS failing.
...
Contributed by Steve Loughran.
Change-Id: Ia9bb84bd6455e210a54cfe9eb944feeda8b58da9
2019-10-10 17:32:25 +01:00
1a77a15fe4
HADOOP-16587. Make ABFS AAD endpoints configurable.
...
Contributed by Bilahari T H.
This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate
in China.
Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c
2019-10-07 13:07:46 +01:00
14cd969b6e
HADOOP-16512. [hadoop-tools] Fix order of actual and expected expression in assert statements
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-10-07 16:38:08 +09:00
b8086bf54d
HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard.
...
Contributed by Steve Loughran.
Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636
2019-10-05 12:52:42 +01:00
6574f27fa3
HADOOP-16570. S3A committers encounter scale issues.
...
Contributed by Steve Loughran.
This addresses two scale issues which has surfaced in large scale benchmarks
of the S3A Committers.
* Thread pools are not cleaned up.
This now happens, with tests.
* OOM on job commit for jobs with many thousands of tasks,
each generating tens of (very large) files.
Instead of loading all pending commits into memory as a single list, the list
of files to load is the sole list which is passed around; .pendingset files are
loaded and processed in isolation -and reloaded if necessary for any
abort/rollback operation.
The parallel commit/abort/revert operations now work at the .pendingset level,
rather than that of individual pending commit files. The existing parallelized
Tasks API is still used to commit those files, but with a null thread pool, so
as to serialize the operations.
Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797
2019-10-04 18:54:22 +01:00
f44abc3e11
HADOOP-16207 Improved S3A MR tests.
...
Contributed by Steve Loughran.
Replaces the committer-specific terasort and MR test jobs with parameterization
of the (now single tests) and use of file:// over hdfs:// as the cluster FS.
The parameterization ensures that only one of the specific committer tests
run at a time -overloads of the test machines are less likely, and so the
suites can be pulled back into the parallel phase.
There's also more detailed validation of the stage outputs of the terasorting;
if one test fails the rest are all skipped. This and the fact that job
output is stored under target/yarn-${timestamp} means failures should
be more debuggable.
Change-Id: Iefa370ba73c6419496e6e69dd6673d00f37ff095
2019-10-04 14:12:31 +01:00
2478cbafe6
YARN-9782. Avoid DNS resolution while running SLS. Contributed by Abhishek Modi.
2019-10-04 14:45:10 +05:30
c99a12167f
HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell.
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-03 22:13:50 -07:00
c19fa3d4fe
HADOOP-16605. Fix testcase testSSLChannelModeConfig
...
Contributed by Sneha Vijayarajan.
2019-10-03 11:13:55 +01:00
559ee277f5
HADOOP-16599. Allow a SignerInitializer to be specified along with a Custom Signer
2019-10-02 16:03:48 -07:00
770adc5d4a
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists
2019-10-01 17:38:11 -07:00
1921e94292
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3
...
Contributed by Steve Loughran.
Includes
-S3A glob scans don't bother trying to resolve symlinks
-stack traces don't get lost in getFileStatuses() when exceptions are wrapped
-debug level logging of what is up in Globber
-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.
-ITestRestrictedReadAccess tests incomplete read access to files.
This adds a builder API for constructing globbers which other stores can use
so that they too can skip symlink resolution when not needed.
Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7
2019-10-01 18:11:05 +01:00
c0edc848a8
HADOOP-16548 : Disable Flush() over config
2019-09-28 20:39:42 -07:00
c89d22d13a
HADOOP-16602. mvn package fails in hadoop-aws.
...
Contributed by Xieming Li.
Follow-up to HADOOP-16445
Change-Id: I72c62d55b734a0f67556844f398ef4a50d9ea585
2019-09-25 14:15:35 +01:00
e346e3638c
HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.
...
Contributed by Steve Loughran.
This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.
This is intended to allow applications to determine
* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.
As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .
The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.
Note
* The remote store is not expected to be checked for the feature;
It is more a check of client API and the client's configuration/knowledge
of the state of the remote system.
* Permissions are not checked.
Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
2019-09-25 12:16:41 +01:00
51c64b357d
HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running
...
This uses the length of the file known at the start of the copy to determine the amount of data to copy.
* If a file is appended to during the copy, the original bytes are copied.
* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,
distcp will now fail. Until now these failures were not detected.
Contributed by Mukund Thakur.
Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884
2019-09-24 11:23:24 +01:00
2b5fc95851
HADOOP-16591 Fix S3A ITest*MRjob failures.
...
Contributed by Siddharth Seth.
Change-Id: I7f08201c9f7c0551514049389b5b398a84855191
2019-09-23 14:58:03 +01:00
aa664d7259
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE ( #1302 ). Contributed by Gabor Bota.
...
Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023
2019-09-23 13:29:01 +02:00
e02b1023c2
HADOOP-16445. Allow separate custom signing algorithms for S3 and DDB ( #1332 )
2019-09-21 11:50:45 +05:30
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 ( #1432 )
...
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
2019-09-20 16:08:30 +05:30
5db32b8ced
HADOOP-16547. make sure that s3guard prune sets up the FS ( #1402 ). Contributed by Steve Loughran.
...
Change-Id: Iaf71561cef6c797a3c66fed110faf08da6cac361
2019-09-18 19:22:15 +02:00
e97f0f1ed9
HADOOP-16565. Region must be provided when requesting session credentials or SdkClientException will be thrown ( #1454 ). Contributed by Gabor Bota.
2019-09-18 10:51:08 +02:00
55ce454ce4
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8.
...
Contributed by Sahil Takiar.
This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common
as the DelegatingSSLSocketFactory and binds the S3A connector to it so that
it can avoid using those HTTPS algorithms which are underperformant on Java 8.
Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd
2019-09-17 11:32:03 +01:00
1505d3f5ff
HADOOP-16566. S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch ( #1433 ). Contributed by Gabor Bota.
...
Change-Id: Ied43ef1522dfc6a1210d6fc58c38d8208824931b
2019-09-12 19:04:57 +02:00
4e273a31f6
HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) ( #1208 ). Contributed by Gabor Bota.
...
Change-Id: I6bbb331b6c0a41c61043e482b95504fda8a50596
2019-09-12 13:12:46 +02:00
9221704f85
HADOOP-16490. Avoid/handle cached 404s during S3A file creation.
...
Contributed by Steve Loughran.
This patch avoids issuing any HEAD path request when creating a file with overwrite=true,
so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile
in their own code.
The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files
for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even
after S3A is patched to not do it itself.
Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy
independently configurable from other retry policies; it is also exponential, but with
different parameters. This is because every HEAD request will refresh any 404 cached in
the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between
attempts to (hopefully) ensure any cached entry wil be gone.
The options and values are:
fs.s3a.s3guard.consistency.retry.interval: 2s
fs.s3a.s3guard.consistency.retry.limit: 7
The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught
so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.
Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:
it will fail fast with a different error message. However, because create(path, overwrite=false) no
longer does HEAD path, we can at least be confident that S3A itself is not creating those cached
404 markers.
Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d
2019-09-11 16:46:25 +01:00
dc9abd27d9
HADOOP-16554. mvn javadoc:javadoc fails in hadoop-aws.
...
Contributed by Xieming Li.
Change-Id: I78e88b5b1ae4702446d2bdd3e2faa3e10b45aef0
2019-09-10 15:05:20 +01:00
147f98629c
HADOOP-16438. ADLS Gen1 OpenSSL config control.
...
Contributed by Sneha Vijayarajan.
Change-Id: Ib79ea6b4a90ad068033e175f3f59c5185868872d
2019-09-09 17:09:32 +01:00
e8ca4fac53
YARN-9812. mvn javadoc:javadoc fails in hadoop-sls. Contributed by Abhishek Modi.
2019-09-07 12:44:40 +05:30
9637097ef9
HDFS-14817. [Dynamometer] Fix start script options parsing which incorrectly interpret options starting with h as a help argument. Contributed by Soya Miyoshi.
2019-09-06 10:24:51 -07:00
ae42c8cb61
HDFS-4819. [Dynamometer] Fix parsing of audit logs which contain = in path names. Contributed by Soya Miyoshi.
2019-09-06 10:13:26 -07:00
511df1e837
HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions
...
Contributed by Steve Loughran.
This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.
For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.
Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
2019-09-05 14:25:15 +01:00
16576fde8e
YARN-9754. Add support for arbitrary DAG AM Simulator. Contributed by Abhishek Modi.
2019-08-29 11:43:40 +05:30
6f068cf53f
HADOOP-16416. mark DynamoDBMetadataStore.deleteTrackingValueMap as final. Contributed by kevin su.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-27 16:17:32 -07:00
567091aa9b
HADOOP-15958. Revisiting LICENSE and NOTICE files.
...
This closes #1307
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2019-08-27 13:47:12 +09:00
23e532d739
Revert "HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran"
...
This reverts commit 69ddb36876
2019-08-26 12:37:26 +02:00
84b1982060
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya.
2019-08-23 14:39:31 +09:00
63c295e298
HDFS-14755. [Dynamometer] Enhance compatibility of Dynamometer with branch-2 builds. Contributed by Takanobu Asanuma.
2019-08-22 09:57:12 -07:00
61b2df2331
HADOOP-16470. Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper.
...
Contributed by Steve Loughran.
Contains HADOOP-16471. Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider.
Change-Id: I06b99b57459cac80bf743c5c54f04e59bb54c2f8
2019-08-22 17:27:56 +01:00
69ddb36876
HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran
2019-08-22 13:56:47 +02:00
ee7c261e1e
HDFS-14763. Fix package name of audit log class in Dynamometer document ( #1335 )
2019-08-22 18:37:16 +09:00
3ad1fcfc8b
YARN-9752. Add support for allocation id in SLS. Contributed by Abhishek Modi
2019-08-21 20:39:51 +05:30
10ec31d20e
YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi.
2019-08-21 13:48:21 +05:30
c765584eb2
HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel ( #919 )
...
* DistCp to support checksum validation when copy blocks in parallel
* address review comments
* add checksums comparison test for combine mode
2019-08-18 18:46:31 -07:00
0e4b757955
HADOOP-16500 S3ADelegationTokens to only log at debug on startup ( #1269 ). Contributed by Steve Loughran.
...
Change-Id: Ifafc15f32791911976d7ebc36fb6e8853f59ed41
2019-08-14 10:50:26 +02:00
274966e675
HDFS-14717. [Dynamometer] Remove explicit search for JUnit dependency JAR from Dynamometer Client as it is packaged in the primary JAR. Contributed by Kevin Su.
2019-08-13 08:52:59 -07:00
189dc10884
HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region. ( #1209 ). Contributed by Steve Loughran.
2019-08-09 17:33:08 +02:00
e25a5c2eab
HADOOP-16499. S3A retry policy to be exponential ( #1246 ). Contributed by Steve Loughran.
2019-08-09 15:52:37 +02:00
43a91f820a
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
...
Contributed by Da Zhou
Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
2019-08-09 12:38:13 +01:00
a92b7a5491
YARN-9694. UI always show default-rack for all the nodes while running SLS.
2019-08-09 11:41:16 +05:30
5840df86d7
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.
...
Contributed by Bilahari T H
Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
2019-08-08 19:08:48 +01:00
b01efe5cf6
HADOOP-16472. findbugs warning on LocalMetadataStore.ttlTimeProvider sync
...
Contributed by Steve Loughran.
Moved the setter and addAncestors to synchronized
Change-Id: Ib362c66d1b8c9124eca7db9a44274ac08d0b3be6
2019-08-02 22:30:48 +01:00
bca86bd289
HADOOP-16469. Update committers.md
...
Contributed by Felipe Lopes.
Change-Id: I5c05b878bde073aeb45bf22340183893f85269e1
2019-07-30 12:47:55 +01:00
7b219778e0
HADOOP-16433. S3Guard: Filter expired entries and tombstones when listing with MetadataStore.listChildren().
...
Contributed by Gabor Bota.
This pulls the tracking of the lastUpdated timestamp of metadata entries up from the DDB metastore into all s3guard stores, and then uses this to filter out expired tombstones from listings.
Change-Id: I80f121236b49c75a024116f65a3ef29d3580b462
2019-07-24 18:11:43 +01:00
a1251addff
HADOOP-16431. Remove useless log in IOUtils.java and ExceptionDiags.java.
...
This closes #1091
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-24 10:04:39 +09:00
4317d33232
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories.
...
Contributed by Steve Loughran and Gabor Bota.
This
* Asks S3Guard to determine the empty directory status.
* Has S3A's root directory rm("/") command to always return false (as abfs does)
* Documents that object stores MAY do this
* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.
It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".
Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108
2019-07-23 14:52:03 +01:00
e60f5e2572
HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun.
2019-07-20 13:11:14 +05:30
0b45293abb
HADOOP-16404. ABFS default blocksize change(256MB from 512MB)
...
Contributed by: Arun Singh
2019-07-19 20:21:28 -07:00
7f1b76ca35
HADOOP-13868. [s3a] New default for S3A multi-part configuration ( #1125 )
2019-07-19 09:49:59 -06:00
cd967c75a7
HADOOP-15847. S3Guard testConcurrentTableCreations to set R/W capacity == 0
...
Contributed by lqjaclee
Change-Id: I4a4d5b29f2677c188799479e4db38f07fa0591d1
2019-07-19 14:46:55 +01:00
d545f9c290
HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise
...
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise
Contributed by: Josh Rosen
2019-07-18 23:19:38 +01:00
c58e11bf52
HADOOP-16383. Pass ITtlTimeProvider instance in initialize method in MetadataStore interface. Contributed by Gabor Bota. ( #1009 )
2019-07-17 16:24:39 +02:00
19a001826f
Revert "HDFS-9913. DistCp to add -useTrash to move deleted files to Trash."
...
Reverting due to test failures if ~/.Trash not present during test setup.
This reverts commit ee3115f488
2019-07-17 13:13:24 +01:00
ee3115f488
HDFS-9913. DistCp to add -useTrash to move deleted files to Trash.
...
Contributed by Shen Yinjie.
Change-Id: I03ac7d22ab1054f8e5de4aa7552909c734438f4a
2019-07-17 11:50:46 +01:00
5672efa5c7
HADOOP-15729. [s3a] Allow core threads to time out. ( #1075 )
2019-07-16 18:14:23 -06:00
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3.
...
Contributed by Steve Loughran
Contains
- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently
This patch doesn't fix the underlying problem but it
* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones
The outcome is that tests should pass consistently and if problems occur we have better diagnostics.
Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
2019-07-12 13:02:25 +01:00
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists.
...
Contributed by Steve Loughran.
This patch
* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.
Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
2019-07-11 18:15:34 +01:00
5043840b1d
HDFS-14410. Make Dynamometer documentation properly compile onto the Hadoop site. Contributed by Erik Krogen.
2019-07-11 23:47:27 +08:00
32925d04d9
HDFS-14640. [Dynamometer] Fix TestDynamometerInfra failure. Contributed by Erik Krogen.
2019-07-11 08:34:39 -07:00
fc0656dd30
HADOOP-16418. [Dynamometer] Fix checkstyle and findbugs warnings. Contributed by Erik Krogen.
2019-07-11 08:29:57 -07:00
c7b5f858a0
HADOOP-16393. S3Guard init command uses global settings, not those of target bucket.
...
Contributed by Steve Loughran.
Change-Id: I226a91ab8d7758340f8d221aa80a7abf9a0d3e8f
2019-07-10 20:57:02 +01:00
90b10a0d54
HDFS-14622. [Dynamometer] Update XML FsImage parsing logic to ignore non-INodeSection entries to fix an issue caused by the presence of Centralized Cache Management functionality. Contributed by Erik Krogen.
2019-07-10 09:59:11 -07:00
738c09349e
HADOOP-16411. Fix javadoc warnings in hadoop-dynamometer.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-07-09 09:44:49 +09:00
de6b7bc67a
HADOOP-16409. Allow authoritative mode on non-qualified paths. Contributed by Sean Mackrory
2019-07-08 19:27:07 +02:00
34747c373f
HADOOP-16396. Allow authoritative mode on a subdirectory. ( #1043 )
2019-07-03 12:04:47 -06:00
ab0b180ddb
HDFS-12345 Add Dynamometer to hadoop-tools, a tool for scale testing the HDFS NameNode with real metadata and workloads. Contributed by Erik Krogen.
2019-06-25 08:07:39 -07:00
366f3deec5
HADOOP-16390. escape javadoc in S3AUtils public methods
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-06-25 17:47:37 +09:00
98d2065643
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 20:17:24 -07:00
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.
...
Contributed by Steve Loughran.
Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
2019-06-20 09:56:40 +01:00
28291a9e8a
HADOOP-16379: S3AInputStream.unbuffer should merge input stream stats into fs-wide stats
...
Contributed by Sahil Takiar
Change-Id: I2bcfaaea00d12c633757069402dcd0b91a5f5c05
2019-06-20 09:42:27 +01:00
450c070a8f
HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations.
...
Contributed by Robert Levas.
This makes the HttpException constructor protected rather than public, so it is possible
to implement custom subclasses of this exception -exceptions which will not be retried.
Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f
2019-06-19 17:43:14 +01:00
1da09bd9d5
HADOOP-16376. ABFS: Override access() to no-op.
...
Contributed by Da Zhou.
Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
2019-06-16 19:20:46 +01:00
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones).
...
Contributed by Gabor Bota.
Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
2019-06-16 17:05:01 +01:00
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours`
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-11 18:40:35 +09:00
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables.
...
Contributed by Steve Loughran
Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
2019-06-07 18:26:10 +01:00
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java
2019-06-07 10:27:41 +09:00
ec26c431f9
HADOOP-16117. Update AWS SDK to 1.11.563.
...
Contributed by Steve Loughran.
Change-Id: I7c46ed2a6378e1370f567acf4cdcfeb93e43fa13
2019-06-06 10:08:18 +01:00
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"
...
This reverts commit b067f8acaa
2019-06-05 13:54:55 +01:00
7724d8031b
Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors"
...
This reverts commit 5906268f0d
2019-06-05 13:54:42 +01:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
0c73dba3a6
HADOOP-16332. Remove S3A dependency on http core.
...
Contributed by Steve Loughran.
Change-Id: I53209c993a405fefdb5e1b692d5a56d027d3b845
2019-05-28 22:50:37 +01:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
5906268f0d
HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors
2019-05-21 11:30:45 -06:00
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.
...
Contributed by Ben Roling.
S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.
You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.
This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.
For a consistent experience, upgrade all clients to use the latest hadoop version.
2019-05-19 22:29:54 +01:00
b067f8acaa
HADOOP-16050: s3a SSL connections should use OpenSSL
...
(cherry picked from commit aebf229c17
2019-05-16 08:57:54 -06:00
c15b3bca86
HADOOP-16294: Enable access to input options by DistCp subclasses.
...
Adding a protected-scope getter for the DistCpOptions, so that a subclass does
not need to save its own copy of the inputOptions supplied to its constructor,
if it wishes to override the createInputFileListing method with logic similar
to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.
Author: Andrew Olson
2019-05-16 16:11:12 +02:00
2d8282bb82
HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu.
2019-05-14 14:05:39 -07:00
ff27e8eabd
HADOOP-16251. ABFS: add FSMainOperationsBaseTest. Re-commit to fix git metadata.
...
Author: Da Zhou
2019-05-10 13:57:30 -07:00
5f1a34649f
Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"
...
This reverts commit 7c2d7c05a9
2019-05-10 13:55:56 -07:00
7c2d7c05a9
HADOOP-16251. ABFS: add FSMainOperationsBaseTest
...
Author: Da Zhou
2019-05-10 12:46:03 -07:00
591ca69823
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
2019-05-09 21:13:29 +01:00
3418bbbb59
HADOOP-16269. ABFS: add listFileStatus with StartFrom.
...
Author: Da Zhou
2019-05-08 17:20:46 +01:00
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.
2019-04-30 11:53:26 +01:00
1cef194a28
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
...
Contributed by Da Zhou.
2019-04-29 13:27:28 +01:00
e1c5ddf2aa
HADOOP-16252. Add prefix to dynamo tables in tests.
...
Contributed by Ben Roling.
2019-04-24 14:55:58 +01:00
a5ceed26f3
HADOOP-16253. Update AssertJ to 3.12.2.
2019-04-16 12:53:52 +09:00
4b4fef2f0e
HADOOP-16227. Upgrade checkstyle to 8.19
2019-04-15 10:46:10 +09:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
bbdbc7a9a1
HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.
2019-04-12 11:52:18 +09:00
cf4efcab3b
HADOOP-16118. S3Guard to support on-demand DDB tables.
...
This is the first step for on-demand operations: things recognize when they are using on-demand tables,
as do the tests.
Contributed by Steve Loughran.
2019-04-11 17:12:12 -07:00
358e928622
YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Addendum. Contributed by Inigo Goiri.
2019-04-09 14:18:05 -07:00
215ffc792e
HADOOP-16197 S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException
...
Contributed by Steve Loughran.
Change-Id: Ie98ca5210bf0009f297edbcacf1fc6dfe5ea70cd.
2019-04-04 21:14:18 +01:00
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
8b6deebb1d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
2019-04-03 19:35:02 +09:00
df578c07ec
HADOOP-16195 MarshalledCredentials toString
...
Change-Id: I4f1bdd2be0d5760c5501dce6edb6122499108b53
2019-03-28 17:01:57 +00:00
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
...
Author: Gabor Bota
2019-03-28 15:59:25 +00:00
cfb0186903
HADOOP-16186. S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren.
...
Author: Gabor Bota
2019-03-28 15:49:56 +00:00
ce4bafdf44
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.
...
Contributed by Siyao Meng
2019-03-26 18:42:54 +00:00
ae2eb2dd42
HADOOP-16201: S3AFileSystem#innerMkdirs builds needless lists ( #636 )
2019-03-22 11:42:00 +00:00
faba3591d3
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.
...
Author: Andrew Olson
2019-03-22 10:35:30 +00:00
9f1c017f44
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-21 11:15:37 +00:00
399563fec6
HDFS-14176. Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia.
2019-03-20 11:20:45 -07:00
568d3ab8b6
HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu.
2019-03-19 09:55:32 +08:00
c0427c84dd
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
2019-03-18 19:13:13 +00:00
1edf1914ac
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file.
...
Contributed by Vishwajeet Dusane.
2019-03-18 13:18:08 +00:00
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
ddede7ae6f
HADOOP-16174. Disable wildfly logs to the console.
...
Follow-on to HADOOP-15851.
Author: Denes Gerencser <dgerencser@cloudera.com>
2019-03-09 18:45:17 +00:00
bb8ad096e7
HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
2019-03-09 09:55:56 -08:00
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-09 16:00:34 +00:00
e0260417ad
HADOOP-16169. ABFS: Bug fix for getPathProperties.
...
Author: Da Zhou <da.zhou@microsoft.com>
2019-03-08 13:53:08 +00:00
05df151d09
HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens.
...
Contributed by Da Zhou.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 14:02:34 +00:00
65f60e56b0
HADOOP-16068. ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store.
...
Contributed by Steve Loughran.
2019-02-28 14:22:32 +00:00
52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util.
...
Contributed by Abhishek Modi
2019-02-26 17:10:41 +00:00
3988e75ca3
HADOOP-16136. ABFS: Should only transform username to short name
...
Contributed by Da Zhou.
2019-02-26 15:37:24 +00:00
a868f59d52
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS.
...
Contributed by Steve Loughran.
2019-02-21 14:08:46 +00:00
aa3ad36605
HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki.
2019-02-20 22:00:57 +09:00
779dae4de7
YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T.
2019-02-20 00:36:57 +05:30
1e0ae6ed15
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
(Revised patch applied after stevel committed the wrong one; that has been reverted)
2019-02-19 11:33:02 +00:00
920a89627d
Revert "HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found."
...
This reverts commit c4a00d1ad3
2019-02-18 14:57:22 +00:00
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR.
2019-02-18 15:21:46 +09:00
134ae8fc80
YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt.
2019-02-14 22:56:52 +05:30
6c999fe4b0
HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by Masatake Iwasaki.
2019-02-12 06:07:47 +09:00
1f1655028e
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-07 21:58:21 +00:00
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
...
(accidentally mixed in two patches)
This reverts commit fa8cd1bf28
2019-02-07 21:57:22 +00:00
546c5d70ef
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on.
2019-02-07 21:48:07 +00:00
de804e53b9
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
2019-02-07 10:07:22 +00:00
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
ba9efe06fa
HADOOP-16074. WASB: Update container not found error code.
...
Contributed by Da Zhou.
2019-02-05 14:39:45 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
02eb91856e
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali.
...
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
2019-01-29 16:50:14 -08:00
3c60303ac5
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document.
2019-01-25 10:27:59 +09:00
00ad9e23e8
HADOOP-16048. ABFS: Fix Date format parser.
...
Contributed by Da Zhou.
2019-01-22 16:40:42 +00:00
30863c5ae3
HADOOP-16044. ABFS: Better exception handling of DNS errors followup
...
Contributed by Da Zhou.
2019-01-14 19:32:56 +00:00
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
c4a00d1ad3
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
2019-01-14 17:27:00 +00:00
04fcbef9c9
HADOOP-16043. NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set.
...
Contributed by Gabor Bota.
2019-01-14 13:12:05 +00:00
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
dddad985d7
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME.
...
Contributed by Da Zhou.
(cherry picked from commit 8b5fbe7a12
2019-01-11 11:13:41 +00:00
852701f793
HADOOP-16036. WASB: Disable jetty logging configuration announcement.
...
Contributed by Da Zhou.
2019-01-10 12:07:09 +00:00
7211269142
HADOOP-15662. Better exception handling of DNS errors.
...
Contributed by Da Zhou.
2019-01-10 12:02:58 +00:00
e8d1900369
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration.
...
Contributed by Da Zhou.
2019-01-10 11:58:39 +00:00
3420e26ae5
HADOOP-16027. [DOC] Effective use of FS instances during S3A integration tests. Contributed by Gabor Bota.
2019-01-09 10:57:58 -07:00
188bebbe7e
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.
...
Contributed by Kai Xie.
2019-01-08 11:57:57 +00:00
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
...
This reverts commit d7152332b3
2019-01-08 14:51:30 +09:00
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
f87b3b11c4
HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu.
2019-01-07 15:56:49 +08:00
1f425271a7
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu."
...
This reverts commit e4fca6aae42b635125fb
2019-01-05 17:35:29 +09:00
040a202b20
HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu.
2019-01-03 21:37:49 +08:00
13f0ee21f2
HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali.
2019-01-02 09:33:09 -07:00
d8f670ff28
HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal.
2018-12-27 08:19:25 -07:00
f82922dcfa
YARN-5168. Added exposed port information for Docker container.
...
Contributed by Xun Liu
2018-12-21 19:44:07 -05:00
a668f8e6c6
HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies.
...
Contributed by Steve Loughran.
2018-12-20 18:09:01 +00:00
784ea3aff6
HADOOP-15965. Upgrade to ADLS SDK which has major performance improvement for ingress/egress.
...
Contributed by Vishwajeet Dusane.
2018-12-20 18:00:05 +00:00
346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException.
...
Contributed by Da Zhou.
2018-12-17 11:15:20 +00:00
62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500.
...
Contributed by Da Zhou.
(cherry picked from commit d09dbcc8fb
2018-12-17 11:10:22 +00:00
b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs.
...
Contributed by Da Zhou.
2018-12-17 11:04:40 +00:00
82b798581d
HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota.
2018-12-12 09:30:13 -07:00
1a25bbe9ec
HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota.
2018-12-11 08:33:13 -07:00
c35de95a22
HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota.
2018-12-11 08:29:39 -07:00
3ff8580f22
HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota)
2018-12-10 14:03:08 -07:00
8fc0d04517
HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-12-07 18:30:36 -08:00
a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS.
...
Contributed by Da Zhou.
2018-12-04 13:39:10 +00:00
7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet.
...
Contributed by Da Zhou.
2018-11-30 10:12:41 +00:00
7eb0d3a324
HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota.
2018-11-29 09:36:39 -07:00
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
2018-11-29 15:12:17 +00:00
085f10e75d
HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
e148c3ff09
HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
5d96b74f33
HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
...
Contributed by Steve Loughran.
2018-11-27 20:39:54 +00:00
300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename.
...
Contributed by Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit 1a3a4960d9
2018-11-27 18:13:07 +00:00
a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2.
...
Contributed by Junhua Gu and Da Zhou.
2018-11-23 14:17:36 +00:00
5ff0cf86a9
HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu.
2018-11-23 13:56:54 +08:00
be0708c6eb
HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu.
2018-11-19 14:00:54 +08:00
914b0cf15f
HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia.
2018-11-15 13:58:13 -08:00
66b1335bb3
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia.
2018-11-15 16:30:24 +09:00
3fade865ce
HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu.
2018-11-14 12:58:57 +08:00
a13be203b7
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
...
Contributed by Da Zhou.
2018-11-13 21:46:18 +00:00
9dbb2b67c6
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console.
...
Author: Da Zhou <da.zhou@microsoft.com>
2018-11-09 11:06:23 +00:00
724c15007b
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries.
...
Contributed by Junhua Gu.
(cherry picked from commit 66715005f9
2018-11-08 17:21:40 +00:00
7574d18538
HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu.
2018-10-26 15:19:56 +08:00
e374584479
HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI
...
(Contributed by Da Zhou via Daniel Templeton)
Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
2018-10-24 14:18:35 -07:00
5e294e950b
YARN-8919. Some tests fail due to NoClassDefFoundError for OperatorCreationException (tasanuma0829 via rkanter)
2018-10-23 13:37:17 -07:00
e2cecb681e
HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-19 13:21:06 -07:00
cc09b2b0c7
HDFS-14007. Incompatible layout when generating FSImage. Contributed by Inigo Goiri.
2018-10-18 17:24:23 -07:00
d59ca43bff
HADOOP-15826. @Retries annotation of putObject() call & uses wrong.
...
Contributed by Steve Loughran and Ewan Higgs.
2018-10-16 20:02:54 +01:00
ef9dc6c44c
HADOOP-15851. Disable wildfly logs to the console.
...
Contributed by Vishwajeet Dusane.
2018-10-15 22:53:55 +01:00
b4a38e7b3e
YARN-8830. SLS tool fix node addition. Contributed by Bibin A Chundatt.
2018-10-15 16:10:25 +05:30
e36ae9639f
HADOOP-15831. Include modificationTime in the toString method of CopyListingFileStatus.
...
Contributed by Ted Yu.
2018-10-12 09:59:19 +01:00
ee816f1fd7
HADOOP-15837. DynamoDB table Update can fail S3A FS init.
...
Contributed by Steve Loughran.
2018-10-11 14:57:38 +01:00
bd50fa956b
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth.
...
Contributed by Da Zhou.
2018-10-09 20:02:12 +01:00
4b5b1ac3d1
MAPREDUCE-7130. Rumen crashes trying to handle MRAppMaster recovery events. Contributed by Peter Bacsko
2018-10-09 13:27:03 -05:00
7ba1cfdea7
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard.
...
Contributed by Gabor Bota
2018-10-09 10:46:41 +01:00
273cc2d4e9
HADOOP-15809. ABFS: better exception handling when making getAccessToken call.
...
Contributed by Da Zhou
2018-10-05 11:29:03 +01:00
7051bd78b1
HADOOP-15795. Make HTTPS the default protocol for ABFS.
...
Contributed by Da Zhou.
2018-10-03 12:52:53 +01:00
4f752d442b
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota
2018-10-03 00:24:29 -07:00
046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota
2018-10-02 21:22:49 -07:00
e8b8604314
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld.
...
Contributed by Abhishek Modi.
2018-10-02 18:57:20 +01:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore.
...
Contributed by Da Zhou
2018-10-02 11:42:52 +01:00
43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization
...
Contributed by Yuan Gao.
2018-10-02 11:37:28 +01:00
a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies.
...
Contributed by Da Zhou.
2018-09-25 20:58:32 +01:00
d5da9928c9
HADOOP-15723. ABFS: Ranger Support.
...
Contributed by Yuan Gao.
2018-09-25 19:13:10 +01:00
2b635125fb
HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu.
2018-09-25 19:48:30 +08:00
c07715e378
HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran
2018-09-24 12:53:21 -07:00
d060cbea48
HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran.
2018-09-24 21:19:47 +05:30
0def61482b
Merge branch 'HADOOP-15407' into trunk
2018-09-22 21:19:12 -06:00
d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read.
...
Contributed by Sneha Varma.
2018-09-21 11:06:24 +01:00
a5692c2da5
HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable.
...
Contributed by Steve Loughran.
2018-09-20 17:36:18 +01:00
8e831ba458
HADOOP-15773. Fixing checkstyle and other issues raised by Yetus.
2018-09-19 16:56:33 -06:00
a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped.
...
Contributed by Steve Loughran
2018-09-19 13:57:39 +01:00
df2166a643
HADOOP-15719. Fail-fast when using OAuth over http.
...
Contributed by Da Zhou.
2018-09-18 12:20:52 +01:00
51d368982b
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution.
...
Contributed by Da Zhou.
2018-09-18 12:09:25 +01:00
524776625d
HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min.
...
Contributed by Da Zhou
2018-09-18 11:48:46 +01:00
1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch.
...
Contributed by Steve Loughran.
2018-09-18 10:56:56 +01:00
26d0c63a1e
HADOOP-15754. s3guard: testDynamoTableTagging should clear existing config.
...
Contributed by Gabor Bota.
2018-09-17 22:40:08 +01:00
b4c23043d3
HADOOP-15757. ABFS: remove dependency on common-codec Base64.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
26211019c8
HADOOP-15753. ABFS: support path "abfs://mycluster/file/path"
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
e5593cbd83
HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts.
...
Contributed by Sean Mackrory.
2018-09-17 19:54:01 +00:00
13c70e9ba3
HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept.
...
Contributed by Sneha Varma.
2018-09-17 19:54:01 +00:00
6801b30733
HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation
...
for storage account with no namespace feature.
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
347a52a866
Fixing findbugs and license issues related to:
...
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
2018-09-17 19:54:01 +00:00
97f06b3fc7
HADOOP-15703. ABFS - Implement client-side throttling.
...
Contributed by Sneha Varma and Thomas Marquardt.
2018-09-17 19:54:01 +00:00
4410eacba7
HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
81dc4a995c
HADOOP-15663. ABFS: Simplify configuration.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
df57c6c3b1
HADOOP-15692. ABFS: extensible support for custom oauth.
...
Contributed by Junhua Gu and Rajeev Bansal.
2018-09-17 19:54:01 +00:00
dd2b22fa31
HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle.
...
Contributed by Thomas Marquardt.
2018-09-17 19:54:01 +00:00
6b6f8cc2be
HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice.
...
Contributed by Sean Mackrory.
2018-09-17 19:54:01 +00:00
9c1e4e8139
HADOOP-15661. ABFS: Add support for ACL.
...
Contributed by Junhua Gu and Da Zhou.
2018-09-17 19:54:01 +00:00
9149b9703e
HADOOP-15660. ABFS: Add support for OAuth
...
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
2018-09-17 19:54:01 +00:00
d6a4f39bd5
HADOOP-15669. ABFS: Improve HTTPS Performance.
...
Contributed by Vishwajeet Dusane.
2018-09-17 19:54:01 +00:00
cc5cc60c41
Fixing issue due to commit 2b2399d6 after rebase onto trunk.
2018-09-17 19:54:01 +00:00
b54b0c1b67
HADOOP-15659. Code changes for bug fix and new tests.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
ce03a93f78
HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests.
...
Contributed by Steve Loughran and Da Zhou.
2018-09-17 19:54:01 +00:00
a271fd0eca
HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
f044deedbb
HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.
...
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
2018-09-17 19:54:01 +00:00
d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran)
2018-09-12 21:04:49 -07:00
d32a8d5d58
HADOOP-14734 add option to tag DDB table(s) created. (Contributed by Gabor Bota and Abe Fine)
2018-09-12 16:36:01 -07:00
1f6c4545cf
HADOOP-15750. Remove obsolete S3A test ITestS3ACredentialsInURL. Contributed by Steve Loughran
2018-09-12 10:58:39 -07:00
47b72c87eb
HADOOP-15635. s3guard set-capacity command to fail fast if bucket is unguarded.
...
Contributed by Gabor Bota.
2018-09-12 09:12:38 -06:00
c44088ac19
YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan.
2018-09-12 16:01:01 +05:30
eb08543c7a
YARN-8103. Add CLI interface to query node attributes. Contributed by Bibin A Chundatt.
2018-09-12 16:01:01 +05:30
89b3ebd11e
YARN-8092. Expose Node Attributes info via RM nodes REST API. Contributed by Weiwei Yang.
2018-09-12 16:01:00 +05:30
87f63b6479
HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran
2018-09-11 17:18:42 -07:00
1eef0acfe5
HADOOP-15677. WASB: Add support for StreamCapabilities.
...
Contributed by Thomas Marquardt.
2018-09-10 14:24:41 +01:00
36c7c78260
HADOOP-15709 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants (Contributed by Gabor Bota)
2018-09-07 10:25:20 -07:00
5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs.
...
Contributed by Steve Loughran.
2018-08-30 14:49:53 +01:00
e8d138ca7c
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out.
...
Contributed by Andras Bokor.
2018-08-30 14:36:00 +01:00
2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length.
...
Contributed by Ewan Higgs
2018-08-30 14:33:16 +01:00
8563fd67be
YARN-8632. Threads in SLS quit without logging exception. Contributed by Xianghao Lu.
2018-08-24 11:23:17 -07:00
96c4575d73
HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore.
2018-08-24 08:14:57 +05:30
d7232857d8
HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (Contributed by Gabor Bota)
2018-08-17 10:15:39 -07:00
0e832e7a74
HADOOP-15642. Update aws-sdk version to 1.11.375.
...
Contributed by Steve Loughran.
2018-08-16 09:58:46 -07:00
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering.
2018-08-16 00:31:59 +09:00
a13929ddcb
HADOOP-15645. ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB. Contributed by Steve Loughran.
2018-08-13 12:57:45 +02:00
da9a39eed1
HADOOP-15583. Stabilize S3A Assumed Role support.
...
Contributed by Steve Loughran.
2018-08-08 22:57:24 -07:00
2ec97abb2e
HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran.
2018-08-08 13:50:23 +02:00
7862f1523f
HADOOP-15400. Improve S3Guard documentation on Authoritative Mode implementation. (Contributed by Gabor Bota)
2018-08-07 20:13:09 -06:00
48673bc2a8
HADOOP-15626. FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
...
Contributed by Steve Loughran.
2018-08-03 16:06:00 -07:00
9fea5c9ee7
YARN-8175. Add support for Node Labels in SLS. Contributed by Abhishek Modi.
2018-07-31 09:36:34 -07:00
0857f116b7
HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu.
2018-07-30 10:53:44 +08:00
59adeb8d7f
HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota.
2018-07-27 18:23:29 -06:00
3d3158cea4
YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang.
2018-07-25 18:59:05 +05:30
45d9568aaa
HADOOP-15547/ WASB: improve listStatus performance.
...
Contributed by Thomas Marquardt.
(cherry picked from commit 749fff577e
2018-07-19 12:31:19 -07:00
752dcce5f4
YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda)
...
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
2018-07-16 10:54:41 -07:00
a08812a1b1
HADOOP-15349. S3Guard DDB retryBackoff to be more informative on limits exceeded. Contributed by Gabor Bota.
2018-07-12 17:24:01 +02:00
d503f65b66
HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting
...
connection in case of timeout.
2018-07-10 17:52:57 +02:00
ca8b80bf59
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan.
...
Contributed by Steve Loughran.
2018-07-10 10:43:59 +01:00
93ac01cb59
HADOOP-15215 s3guard set-capacity command to fail on read/write of 0 (Gabor Bota)
2018-07-03 13:50:11 -07:00
1804a31515
HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-07-02 10:48:20 -07:00
73746c5da7
HDFS-13707. [PROVIDED Storage] Fix failing integration tests in ITestProvidedImplementation. Contributed by Virajith Jalaparti.
2018-06-29 09:56:13 -07:00
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.
2018-06-28 14:37:22 +09:00
7a3c6e9c3c
HADOOP-15550. Avoid static initialization of ObjectMappers
2018-06-25 15:36:45 -07:00
c687a6617d
HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota.
2018-06-25 14:59:41 -06:00
e16e5b307d
HDDS-177. Create a releasable ozonefs artifact
...
Contributed by Marton, Elek.
2018-06-24 01:05:04 -07:00
ca14fec02f
HDDS-184. Upgrade common-langs version to 3.7 in hadoop-tools/hadoop-ozone.
...
Contributed by Takanobu Asanuma.
2018-06-23 15:49:44 -07:00
55fad6a3de
HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota.
2018-06-22 11:36:56 -06:00
b089a06793
HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota.
2018-06-20 16:45:08 -06:00
f34744603e
HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii
2018-06-17 23:12:18 -07:00
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
f4c7c91123
HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan.
2018-06-13 00:36:02 -07:00
c35481594f
YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola.
2018-06-12 10:59:50 -07:00
d901be6795
HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks.
...
Contributed by Esfandiar Manii.
2018-06-06 18:28:14 +01:00
3b34148c4f
HDDS-88. Create separate message structure to represent ports in DatanodeDetails.
...
Contributed by Nanda Kumar.
2018-05-30 08:52:07 -07:00
5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota)
2018-05-29 19:20:22 -07:00
30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota.
2018-05-29 13:38:15 -07:00
8fdc993a99
HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu.
2018-05-28 16:45:42 -07:00
d14e26b31f
HADOOP-15477. Make unjar in RunJar overrideable
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-28 17:29:59 +09:00
ba842847c9
HADOOP-15478. WASB: hflush() and hsync() regression.
...
Contributed by Thomas Marquardt.
2018-05-21 11:12:34 +01:00
cc3600aabd
HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest.
...
Contributed by Steve Loughran.
(cherry picked from commit 170f1040d4
2018-05-17 19:08:55 +01:00
b670837469
HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name.
...
Contributed by Sean Mackrory.
2018-05-15 16:13:56 +01:00
ba051b0686
HADOOP-15354. hadoop-aliyun & hadoop-azure modules to mark hadoop-common as provided
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-10 18:38:08 +09:00
46e0f27862
HDDS-18. Ozone Shell should use RestClient and RpcClient. Contributed by Lokesh Jain.
2018-05-09 23:00:13 -07:00
1cfe7506f7
HADOOP-15356. Make HTTP timeout configurable in ADLS connector. Contributed by Atul Sikaria and Sean Mackrory.
2018-05-09 15:15:51 -06:00
343b51dcf4
HDDS-28. Ozone:Duplicate declaration in hadoop-tools/hadoop-ozone/pom.xml. Contributed by Tsz Wo Nicholas Sze.
2018-05-09 20:20:21 +05:30
208b97e969
HDDS-20. Ozone: Add support for rename key within a bucket for rpc client. Contributed by Lokesh Jain.
2018-05-09 19:06:07 +05:30
8981674bbc
HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor Bota)
2018-05-08 18:58:42 -07:00
69aac696d9
HADOOP-13649 s3guard: implement time-based (TTL) expiry for LocalMetadataStore (Gabor Bota)
2018-05-08 15:29:54 -07:00
5b11b9fd41
HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication.
...
Contributed by Thomas Marquardt
2018-05-07 11:54:08 +01:00
96c843f64b
HADOOP-15444 ITestS3GuardToolDynamo should only run with -Ddynamo (Aaron Fabbri)
2018-05-04 11:34:45 -07:00
85381c7b60
HADOOP-15434. Upgrade to ADLS SDK that exposes current timeout.
2018-05-02 21:30:31 -06:00
919865a34b
HADOOP-15239 S3ABlockOutputStream.flush() be no-op when stream closed. Contributed by Gabor Bota.
2018-04-30 16:02:57 -07:00
ba2db39c65
YARN-8152. Add chart in SLS to illustrate the throughput of the scheduler. Contributed by Tao Yang.
2018-04-27 15:36:21 +08:00
7d8bcf534a
HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash. (Gabor Bota)
2018-04-26 20:41:29 -07:00
2adda92de1
HDFS-7240 git commit fixup
...
Signed-off-by: Owen O'Malley <omalley@apache.org>
2018-04-26 13:52:17 -07:00
ae8ac7f082
HDFS-13424. Ozone: Refactor MiniOzoneClassicCluster. Contributed by Nanda Kumar.
2018-04-26 05:36:04 -07:00
025058f251
HDFS-13446. Ozone: Fix OzoneFileSystem contract test failures. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
8b832f3c35
HDFS-13405. Ozone: Rename HDSL to HDDS.
...
Contributed by Ajay Kumar, Elek Marton, Mukul Kumar Singh, Shashikant Banerjee and Anu Engineer.
2018-04-26 05:36:04 -07:00
7ae9543a57
HDFS-13391. Ozone: Make dependency of internal sub-module scope as provided in maven. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
04e1287771
HDFS-13298. Ozone: Make ozone/hdsl/cblock modules turned off by default. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
4e61bc431e
HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects.
...
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
2018-04-26 05:36:04 -07:00
5e7164c614
HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key.
...
Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
61651dcf5c
HDFS-13108. Ozone: OzoneFileSystem: Simplified url schema for Ozone File System. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
cd807c9890
HDFS-13133. Ozone: OzoneFileSystem: Calling delete with non-existing path shouldn't be logged on ERROR level. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
9272e1021d
HDFS-12636. Ozone: OzoneFileSystem: Implement seek functionality for rpc client. Contributed by Lokesh Jain.
2018-04-26 05:36:04 -07:00
4460ac9c57
HDFS-12756. Ozone: Add datanodeID to heartbeat responses and container protocol. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
740a06cdd7
HDFS-12719. Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
e395244e95
HDFS-12758. Ozone: Correcting assertEquals argument order in test cases. Contributed by Bharat Viswanadham.
2018-04-26 05:36:04 -07:00
2ec24a7542
HDFS-12774. Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
895a4cc60a
HDFS-12632. Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
d90ab3365d
HDFS-12608. Ozone: Remove Warnings when building. Contributed by Bharat Viswanadham.
2018-04-26 05:36:04 -07:00
9445a9267f
HDFS-12572. Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
0b3270a8de
HDFS-12513. Ozone: Create UI page to show Ozone configs by tag. Contributed by Ajay Kumar.
2018-04-26 05:36:04 -07:00
5e4a6b686c
HDFS-12425. Ozone: OzoneFileSystem: read/write/create/open/getFileInfo APIs. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
f2d9360b9d
HDFS-12199. Ozone: OzoneFileSystem: OzoneFileystem initialization code. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
3e2ca23a53
HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
989a3929a9
HADOOP-14756 S3Guard: expose capability query in MetadataStore and add tests of authoritative mode (Gabor Bota)
2018-04-23 15:26:53 -07:00
fd24fd0ff7
YARN-8137. Parallelize node addition in SLS. Contributed by Abhishek Modi.
2018-04-20 09:09:12 -07:00
1312f9ae4c
HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules.
2018-04-18 16:04:00 +08:00
78860372bd
YARN-8134. Support specifying node resources in SLS. Contributed by Abhishek Modi.
2018-04-17 13:11:34 -07:00
b0aff8a962
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran.
2018-04-10 23:55:38 -07:00
7c9cdad6d0
HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo.
2018-04-10 21:31:48 -07:00
ea3849f0cc
HADOOP-14759 S3GuardTool prune to prune specific bucket entries. Contributed by Gabor Bota.
2018-04-05 20:23:17 -07:00
5a174f8ac6
HADOOP-14758. S3GuardTool.prune to handle UnsupportedOperationException.
...
Contributed by Gabor Bota.
2018-04-03 16:31:34 +01:00
6542d17ea4
HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu.
2018-03-30 20:23:05 +08:00
9d7a9031a5
HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria.
2018-03-29 10:13:40 -06:00
081c350188
HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao
2018-03-28 11:58:59 -07:00
fcea5a4d79
YARN-7794. SLSRunner is not loading timeline service jars, causing failure. (Yufei Gu via Haibo Chen)
2018-03-26 14:30:11 -07:00
82665a7887
HADOOP-15299. Bump Jackson 2 version to Jackson 2.9.x.
2018-03-26 14:05:15 -06:00
7ce6b41509
HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.
2018-03-20 21:12:20 -07:00
d67a5e2dec
HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu.
2018-03-19 15:02:37 +08:00
f47659fb97
MAPREDUCE-7047. Make HAR tool support IndexedLogAggregtionController. (Xuan Gong via wangda)
...
Change-Id: Ice5ae1c706f2476361997dcbb29f2c33c77d4f0c
2018-03-15 13:26:01 -07:00
1976e0066e
HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories.
...
Contributed by Steve Loughran.
2018-03-15 18:05:14 +00:00
5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-15 16:12:07 +09:00
45cccadd2e
HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang
2018-03-13 11:08:11 -07:00
45d1b0fdcc
HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer
2018-03-12 20:05:39 -07:00
dd05871b8b
HADOOP-15297. Make S3A etag => checksum feature optional.
...
Contributed by Steve Loughran.
2018-03-12 14:01:42 +00:00
7ef4d942dd
HADOOP-15273.distcp can't handle remote stores with different checksum algorithms.
...
Contributed by Steve Loughran.
2018-03-08 11:24:06 +00:00
3bd6b1fd85
HADOOP-15292. Distcp's use of pread is slowing it down.
...
Contributed by Virajith Jalaparti.
2018-03-08 11:15:46 +00:00
e0307e53e2
HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled.
...
Contributed by Anis Elleuch.
(cherry picked from commit 1dedc68f9d
2018-03-07 18:51:10 +00:00
2e1e049bd0
HADOOP-15288. TestSwiftFileSystemBlockLocation doesn't compile.
...
Contributed by Steve Loughran.
2018-03-05 18:12:38 +00:00
8110d6a0d5
HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions.
...
Contributed by Aaron Fabbri.
2018-03-05 14:06:20 +00:00
e99a1e7767
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-02 16:15:04 -08:00
923e1773e1
HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang)
2018-03-01 15:36:10 -08:00
29233c3c4a
HADOOP-15274. Move hadoop-openstack to slf4j
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-01 15:08:30 -08:00
edc9f1451b
HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake.
...
Contributed by Steve Loughran
2018-02-28 14:42:25 +00:00
7af4f34de5
YARN-7929. Support to set container execution type in SLS. (Jiandan Yang via Weiwei Yang)
2018-02-28 17:58:09 +08:00
2fa7963c3d
HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi.
2018-02-24 14:41:55 -08:00
68ce193efc
MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák
2018-02-23 14:37:26 -08:00
84cea0011f
YARN-7732. Support Generic AM Simulator from SynthGenerator. (Contributed by Young Chen via curino)
2018-02-20 17:00:34 -08:00
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-02-18 22:19:39 +09:00
7ac88244c5
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key.
...
Contributed by Steve Loughran.
2018-02-16 16:37:06 +00:00
481d79fedc
HADOOP-13972. ADLS to support per-store configuration.
...
Contributed by Sharad Sonker.
(cherry picked from commit 050f5287b7
2018-02-15 16:27:31 +00:00
9a013b255f
HADOOP-15176. Enhance IAM Assumed Role support in S3A client.
...
Contributed by Steve Loughran
(cherry picked from commit 96c047fbb9
2018-02-15 15:57:10 +00:00
b27ab7dd81
HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document.
...
Contributed by Steve Loughran.
(cherry picked from commit c761e658f6
2018-02-15 14:57:56 +00:00
c9a373fb14
HADOOP-15090. Add ADL troubleshooting doc.
...
Contributed by Steve Loughran.
(cherry picked from commit 58a2120e8a
2018-02-15 14:26:00 +00:00
f20dc0d577
HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions.
...
Contributed by Andras Bokor.
2018-02-14 16:20:14 +00:00
8cf88fcd1f
HADOOP-15187. Remove ADL mock test dependency on REST call invoked from Java SDK.
...
Contributed by Vishwajeet Dusane.
(cherry picked from commit bd5ab59125
2018-02-12 15:13:00 +00:00
60f9e60b3b
Preparing for 3.2.0 development
...
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
4304fcd5bd
HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen.
2018-02-06 13:43:45 -08:00
d4e13a4647
HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh.
2018-02-02 11:37:51 -08:00
9b81cb0537
YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang.
2018-01-31 01:30:17 -08:00
38af237969
YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh)
2018-01-31 01:30:17 -08:00
7fd287b4af
HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line.
...
Contributed by Vishwajeet Dusane.
(cherry picked from commit 95a96b13e2
2018-01-29 09:48:14 -08:00
ff8378eb1b
YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang
2018-01-25 17:47:19 -06:00
16be42d309
YARN-7798. Refactor SLS Reservation Creation. Contributed by Young Chen.
2018-01-25 13:06:01 -08:00
f63d13f10d
HADOOP-15185. Update adls connector to use the current version of ADLS SDK.
...
Contributed by Atul Sikaria.
2018-01-23 09:09:11 -08:00
1093a73689
HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
...
Contributed by Aaron Fabbri
2018-01-18 13:13:58 +00:00
f274fe33ea
Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits."
...
This reverts commit 35ad9b1dd2
2018-01-18 12:35:57 +00:00
268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran.
2018-01-17 00:05:24 -08:00
9195a6e302
HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)
2018-01-17 15:55:59 +08:00
a0c71dcc33
HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing
...
after OutputCommitter patch.
Contributed by Steve Loughran
2018-01-15 11:33:32 +00:00
2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-01-15 15:45:51 +09:00
1a09da7400
HADOOP-15163. Fix S3ACommitter documentation
...
Contributed by Alessandro Andrioni.
(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
2018-01-10 15:37:07 +00:00
b62a5ece95
HADOOP-15161. s3a: Stream and common statistics missing from metrics
...
Contributed by Sean Mackrory
2018-01-09 18:46:52 +00:00
52babbb4a0
HADOOP-15086. NativeAzureFileSystem file rename is not atomic.
...
Contributed by Thomas Marquardt
2017-12-22 11:39:55 +00:00
c8ff0cc304
HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls.
...
Contributed by Steve Loughran
2017-12-21 14:58:58 +00:00
ef450df443
HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance.
...
Contributed by Steve Loughran.
2017-12-21 14:15:53 +00:00
5ab632baf5
HDFS-12949. Fix findbugs warning in ImageWriter.java.
2017-12-21 10:04:34 +09:00
1ba491ff90
HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive
2017-12-20 18:25:33 +00:00
35ad9b1dd2
HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
...
Contributed by Aaron Fabbri
2017-12-18 21:19:06 +00:00
4b3a785914
HDFS-12903. [READ] Fix closing streams in ImageWriter. Contributed by Virajith Jalaparti
2017-12-15 17:51:42 -08:00
e515103a83
Revert "HDFS-12903. [READ] Fix closing streams in ImageWriter"
...
This reverts commit c1bf2654b0
2017-12-15 17:51:42 -08:00
8239e3afb3
HDFS-12712. [9806] Code style cleanup
2017-12-15 17:51:41 -08:00
80c3fec3a1
HDFS-12912. [READ] Fix configuration and implementation of LevelDB-based alias maps
2017-12-15 17:51:41 -08:00
c89b29bd42
HDFS-12893. [READ] Support replication of Provided blocks with non-default topologies.
2017-12-15 17:51:41 -08:00
0f6aa9564c
HDFS-12905. [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage.
2017-12-15 17:51:41 -08:00
962b5e722b
HDFS-12903. [READ] Fix closing streams in ImageWriter
2017-12-15 17:51:41 -08:00
4531588a94
HDFS-11640. [READ] Datanodes should use a unique identifier when reading from external stores
2017-12-15 17:51:41 -08:00
fb996a32a9
HDFS-12894. [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update
2017-12-15 17:51:40 -08:00
9c35be86e1
HDFS-12713. [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata. Contributed by Ewan Higgs
2017-12-15 17:51:40 -08:00
a027055dd2
HDFS-12885. Add visibility/stability annotations. Contributed by Chris Douglas
2017-12-15 17:51:40 -08:00
352f994b64
HDFS-12665. [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb). Contributed by Ewan Higgs.
2017-12-15 17:51:40 -08:00
4d59dabb7f
HDFS-12809. [READ] Fix the randomized selection of locations in {{ProvidedBlocksBuilder}}.
2017-12-15 17:51:40 -08:00
3d3be87e30
HDFS-12778. [READ] Report multiple locations for PROVIDED blocks
2017-12-15 17:51:39 -08:00
3b1d30301b
HDFS-12775. [READ] Fix reporting of Provided volumes
2017-12-15 17:51:39 -08:00
6cd80b2521
HDFS-12779. [READ] Allow cluster id to be specified to the Image generation tool
2017-12-15 17:51:39 -08:00
90d1b47a2a
HDFS-12776. [READ] Increasing replication for PROVIDED files should create local replicas
2017-12-15 17:51:39 -08:00
87dc026bee
HDFS-12789. [READ] Image generation tool does not close an opened stream
2017-12-15 17:51:39 -08:00
c293cc8e9b
HDFS-12671. [READ] Test NameNode restarts when PROVIDED is configured
2017-12-15 17:51:39 -08:00
71d0a82571
HDFS-12607. [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
2017-12-15 17:51:39 -08:00
98f5ed5aa3
HDFS-11902. [READ] Merge BlockFormatProvider and FileRegionProvider.
2017-12-15 17:51:38 -08:00
d6a9a89973
HDFS-12605. [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase
2017-12-15 17:51:38 -08:00
17052c4aff
HDFS-12584. [READ] Fix errors in image generation tool from latest rebase
2017-12-15 17:51:38 -08:00
aca023b72c
HDFS-12289. [READ] HDFS-12091 breaks the tests for provided block reads
2017-12-15 17:51:38 -08:00
546b95f484
HDFS-11673. [READ] Handle failures of Datanode with PROVIDED storage
2017-12-15 17:51:38 -08:00
4851f06bc2
HDFS-11791. [READ] Test for increasing replication of provided files.
2017-12-15 17:51:38 -08:00
aa5ec85f7f
HDFS-11663. [READ] Fix NullPointerException in ProvidedBlocksBuilder
2017-12-15 17:51:37 -08:00
d65df0f273
HDFS-11190. [READ] Namenode support for data stored in external stores.
2017-12-15 17:51:37 -08:00
8da3a6e314
HDFS-10706. [READ] Add tool generating FSImage from external store
2017-12-15 17:51:37 -08:00
f86c81d923
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu
2017-12-14 11:14:52 +08:00
28792b6b7f
Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu"
...
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
2017-12-14 11:05:11 +08:00
00129c5314
HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani.
2017-12-12 00:28:02 +05:30
ce04340ec7
HADOOP-15104. AliyunOSS: change the default value of max error retry. Contributed by Jinhu Wu
2017-12-08 23:03:54 +08:00
73b86979d6
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu
2017-12-06 12:01:36 +08:00
6555af81a2
HADOOP-14475 Metrics of S3A don't print out when enabled. Contributed by Younger and Sean Mackrory.
2017-12-05 11:06:32 -08:00
3150c019ae
HADOOP-15071 S3a troubleshooting docs to add a couple more failure modes.
...
Contributed by Steve Loughran
2017-12-05 15:05:41 +00:00
0ea182d0fa
HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.
...
Contributed by Rajesh Balamohan
2017-11-28 11:52:59 +00:00
3cd75845da
MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran.
2017-11-27 22:01:00 +05:30
de8b6ca5ef
HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints.
...
Contributed by Steve Loughran and Ryan Blue.
2017-11-22 15:28:12 +00:00
c326fc89b0
HADOOP-15024 Support user agent configuration and include that & Hadoop version information to oss server.
...
Contributed by Sammi Chen.
2017-11-20 18:56:42 +00:00
6903cf096e
HADOOP-13514. Upgrade maven surefire plugin to 2.20.1
...
Signed-off-by: Allen Wittenauer <aw@apache.org>
2017-11-19 12:39:37 -08:00
18621af7ae
HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. Contributed Genmao Yu
2017-11-14 17:58:37 +08:00
ba8136615a
YARN-7413. Support resource type in SLS (Contributed by Yufei Gu via Daniel Templeton)
...
Change-Id: Ic0a897c123c5d2f57aae757ca6bcf1dad7b90d2b
2017-11-09 12:09:48 -08:00
bf6a660232
HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge.
2017-11-09 10:16:12 -08:00
f2df6b8983
HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru).
2017-11-08 18:07:12 -08:00
53c0fb7efe
HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh)
2017-11-02 17:15:33 -07:00
cc3f3eca40
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi.
2017-11-02 18:32:24 +09:00
139cc752d2
HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li
2017-10-27 21:59:27 -07:00
99880d0a16
YARN-6505. Define the strings used in SLS JSON input file format. (Contributed by Gergely Novak)
2017-10-27 14:41:37 -07:00
625039ef20
HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru).
2017-10-25 15:51:27 -07:00
281d83604d
YARN-7318. Fix shell check warnings of SLS. (Gergely Novák via wangda)
2017-10-20 15:54:15 -07:00
f36cbc8475
HADOOP-14942. DistCp#cleanup() should check whether jobFS is null.
...
Contributed by Andras Bokor.
2017-10-20 22:27:04 +01:00
46eb1033a8
YARN-6546. SLS is slow while loading 10k queues. (Yufei Gu via Haibo Chen)
2017-10-17 16:04:19 -07:00
9fcc3a1fc8
HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled.
...
Contributed by Santhosh G Nayak
2017-10-16 16:01:47 +01:00
3de574413c
HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB.
...
Contributed by Varada Hemeswari.
2017-10-11 18:06:43 +01:00
572cdb5463
HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB
...
Contributed by Kannapiran Srinivasan.
2017-10-06 17:43:38 +01:00
c071aad5da
YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen)
2017-10-05 15:50:13 -07:00
9288206cb3
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
...
Contributed by Sivaguru Sankaridurg
2017-10-05 15:05:23 +01:00
79e37dc6db
YARN-7211. AMSimulator in SLS does't work due to responseId refactoring.(Contributed by Botong Huang via Yufei Gu)
2017-10-03 15:10:50 -07:00
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari.
2017-10-04 07:01:54 +09:00
8aca46e53f
HADOOP-14651. Update okhttp version to 2.7.5. Contributed by Ray Chiang and John Zhuge.
2017-09-29 12:15:06 -07:00
a530e7ab3b
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB
...
Contributed by Varada Hemeswari
2017-09-28 19:52:56 +01:00
47011d7dd3
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran
2017-09-25 15:59:38 -07:00
bilaharith
Gabor Bota
Sneha Vijayarajan
Jeetesh Mangwani
Steve Loughran
Andras Bokor
Takanobu Asanuma
Phil Zampino
Yuming Wang
Da Zhou
Steve Loughran
Ashvin
Szilard Nemeth
lqjacklee
bilaharith
pingsutw
Abhishek Modi
Stephen O'Donnell
Siddharth Seth
Xieming Li
Mukund Thakur
Vinayakumar B
Sahil Takiar
Erik Krogen
Akira Ajisaka
Ewan Higgs
Ewan Higgs
bibinchundatt
KAI XIE
Da Zhou
Abhishek Modi
Felipe Lopes
sunlisheng
Ayush Saxena
Arun Singh
Sean Mackrory
Josh Rosen
Shen Yinjie
Yiqun Lin
Masatake Iwasaki
kkori
Robert Levas
Wanqiang Ji
Ben Roling
Andrew Olson
Weiwei Yang
DadanielZ
Aaron Fabbri
Rajat Khandelwal
Giovanni Matteo Fumarola
Ben Roling
Siyao Meng
Lokesh Jain
Ajay Kumar
Adam Antal
Vishwajeet Dusane
Denes Gerencser
Shweta Yakkali
Abhishek Modi
Ranith Sardar
Shweta Yakkali
Sean Mackrory
Sunil G
Sean Mackrory
Eric Yang
Inigo Goiri
Arpit Agarwal
Sammi Chen
Junhua Gu
Daniel Templeton
Robert Kanter
Ted Yu
Jason Lowe
Mingliang Liu
Thomas Marquardt
Naganarasimha
Yufei Gu
Surendra Singh Lilhore
Wangda Tan
Todd Lipcon
Anu Engineer
Chris Douglas
Jitendra Pandey
Johan Gustavsson
Steve Loughran
Mukul Kumar Singh
Owen O'Malley
Nanda kumar
Chen Liang
Xiaoyu Yao
Xiao Chen
Haibo Chen
fang zhenyi
Gergely Novák
Ray Chiang
Xuan Gong
Carlo Curino
Arun Suresh
Yufei Gu
Mukul Kumar Singh
Virajith Jalaparti
Kai Zheng
Rohith Sharma K S
Kai Zheng
John Zhuge
Subru Krishnan
John Zhuge