d545f9c290
HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise
...
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise
Contributed by: Josh Rosen
2019-07-18 23:19:38 +01:00
b4466a3b0a
HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
...
Contributed by Gopal V and Atilla Magyar.
Change-Id: I066d5eece332a1673594de0f9b484443f95530ec
2019-07-17 13:50:02 +01:00
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3.
...
Contributed by Steve Loughran
Contains
- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently
This patch doesn't fix the underlying problem but it
* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones
The outcome is that tests should pass consistently and if problems occur we have better diagnostics.
Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
2019-07-12 13:02:25 +01:00
8fb5ca3f40
HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.
2019-07-11 13:23:25 -07:00
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists.
...
Contributed by Steve Loughran.
This patch
* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.
Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
2019-07-11 18:15:34 +01:00
aa9f0850e8
HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena.
2019-07-02 13:22:30 -07:00
129576f628
HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian.
2019-06-24 12:09:17 -07:00
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar.
2019-06-24 11:59:07 -07:00
b52fd05d42
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma.
2019-06-24 22:03:04 +05:30
ba221dc50e
HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma.
2019-06-24 22:03:04 +05:30
559cb11551
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails.
2019-06-24 22:03:03 +05:30
f40e10b349
HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar.
2019-06-24 22:03:03 +05:30
371452e260
HADOOP-15989. Synchronized at CompositeService#removeService is not required. Contributed by Prabhu Joseph.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-21 17:05:13 -07:00
840d02ca5b
HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 19:57:48 -07:00
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.
...
Contributed by Steve Loughran.
Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
2019-06-20 09:56:40 +01:00
3ab77d9bc9
HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao.
2019-06-18 09:58:29 -07:00
dd4a7633ec
HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 23:47:57 -07:00
56c621c68f
HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 22:56:00 -07:00
f1c239c6a4
HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor.
2019-06-17 21:51:33 -07:00
10311c30b0
HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong.
2019-06-17 19:55:56 -07:00
1e92db5a1e
HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend.
2019-06-17 18:29:34 -07:00
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones).
...
Contributed by Gabor Bota.
Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
2019-06-16 17:05:01 +01:00
076618677d
HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm.
2019-06-14 20:05:20 -07:00
c7554ffd5c
HADOOP-16373. Fix typo in FileSystemShell#test documentation ( #968 )
2019-06-14 14:36:04 -07:00
54f9f75a44
HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles.
2019-06-14 10:17:25 +09:00
4ea6c2f457
HADOOP-16354. Enable AuthFilter as default for WebHDFS.
...
Contributed by Prabhu Joseph
2019-06-11 18:41:08 -04:00
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours`
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-11 18:40:35 +09:00
d6d95d2686
HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 10:23:18 -07:00
76b94c274f
HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen.
2019-06-07 14:20:44 -07:00
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables.
...
Contributed by Steve Loughran
Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
2019-06-07 18:26:10 +01:00
14552d19e3
HADOOP-16344. Make DurationInfo public unstable.
...
Second attempt: imports fixed up.
Contributed by Huan-Ping Su and Kai Xie.
Change-Id: Ib2a85dd4b2d12b840692cc854cc53ddb3bdde7d5
2019-06-07 12:55:17 +01:00
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java
2019-06-07 10:27:41 +09:00
1f244b4227
Revert "HADOOP-16344. Make DurationInfo public unstable."
...
This reverts commit 829848ba2e
2019-06-06 17:49:31 +01:00
829848ba2e
HADOOP-16344. Make DurationInfo public unstable.
...
Contributed by Huan-Ping Su
Change-Id: I64a94cf382f9db78b4ef49a3912f25f0d906af7c
2019-06-06 12:52:49 +01:00
294695dd57
HADOOP-16314. Make sure all web end points are covered by the same authentication filter.
...
Contributed by Prabhu Joseph
2019-06-05 18:55:13 -04:00
d1aad44490
HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.
2019-06-05 21:33:00 +08:00
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"
...
This reverts commit b067f8acaa
2019-06-05 13:54:55 +01:00
7724d8031b
Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors"
...
This reverts commit 5906268f0d
2019-06-05 13:54:42 +01:00
827a84778a
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer -follow-on patch.
...
This follow-on patch to HADOOP-16266 fixes up the problem where logs were being full of
stack traces because the timeout passed down to select was in nanos, whereas the API
expected millis.
Contributed by Erik Krogen.
Change-Id: I5c6e9ddf68127b1d7e0ca0e179d036eb9941e445
2019-06-04 15:56:47 +01:00
6f5a36c13c
HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:21:58 -07:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
93d7630062
HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-24 18:42:46 +02:00
f96a2df38d
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
2019-05-23 10:28:37 -07:00
ea0b1d8fba
HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation.
...
Contributed by Prabhu Joseph
2019-05-23 11:36:32 -04:00
a771e2a638
HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang.
2019-05-23 13:59:42 +09:00
5906268f0d
HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors
2019-05-21 11:30:45 -06:00
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.
...
Contributed by Ben Roling.
S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.
You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.
This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.
For a consistent experience, upgrade all clients to use the latest hadoop version.
2019-05-19 22:29:54 +01:00
4cb3da6ac7
HADOOP-16248. MutableQuantiles leak memory under heavy load.
...
Contributed by Alexis Daboville,
2019-05-17 15:15:22 +01:00
b067f8acaa
HADOOP-16050: s3a SSL connections should use OpenSSL
...
(cherry picked from commit aebf229c17
2019-05-16 08:57:54 -06:00
2713dcf6e9
HADOOP-16307. Intern User Name and Group Name in FileStatus.
...
Author: David Mollitor
2019-05-16 16:02:07 +02:00
d4c8858586
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy.
2019-05-15 17:41:36 -07:00
389e640f0c
HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao.
2019-05-13 11:46:16 -07:00
f257497b0f
HADOOP-16299. [JDK 11] Build fails without specifying -Djavac.version=11
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-09 14:49:46 +09:00
96dc5cedfe
HADOOP-16293. AuthenticationFilterInitializer doc has speudo instead of pseudo.
...
Author: Prabhu Joseph
2019-05-08 10:18:20 +01:00
713e8a27ae
HADOOP-16238. Add the possbility to set SO_REUSEADDR in IPC Server Listener. Contributed by Peter Bacsko.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-07 17:48:27 -07:00
93f2283a69
HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-06 15:47:33 -07:00
f1875b205e
HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena.
2019-05-03 11:22:14 +05:30
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
4877f0aa51
HDFS-3246: pRead equivalent for direct read path ( #597 )
...
HDFS-3246: pRead equivalent for direct read path
Contributed by Sahil Takiar
2019-04-30 14:52:16 -07:00
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.
2019-04-30 11:53:26 +01:00
a703dae25e
HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.
2019-04-24 10:39:00 -06:00
f4ab9370f5
HADOOP-16026:Replace incorrect use of system property user.name.
...
Contributed by Dinesh Chitlangia.
2019-04-22 14:02:13 -07:00
fb1c549139
HDFS-14374. Expose total number of delegation tokens in AbstractDelegationTokenSecretManager. Contributed by CR Hota.
2019-04-22 13:32:08 -07:00
1ddb48872f
HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin.
2019-04-22 08:16:57 -07:00
008766c119
HADOOP-15014. KMS should log the IP address of the clients. Contributed by Zsombor Gegesy.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-16 05:28:18 -07:00
b1120d27ab
HADOOP-16249. Make CallerContext LimitedPrivate scope to Public.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-16 11:18:21 +09:00
1943db5571
HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre.
...
Author: Gabor Bota <gabor.bota@cloudera.com>
2019-04-12 18:28:38 -07:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
626fec652b
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-04-12 17:37:51 -07:00
87407553ef
HADOOP-16243. Change Log Level to trace in NetUtils.java. Contributed by chencan.
2019-04-10 13:21:04 -07:00
813cee1a18
HDFS-14420. Fix typo in KeyShell console. Contributed by Hu Xiaodong.
2019-04-10 11:18:40 -07:00
65deb1ac42
HADOOP-16179. hadoop-common pom should not depend on kerb-simplekdc
...
The hadoop-common pom currently has a dependency on kerb-simplekdc. In
fact, the only classes used from Kerby are in kerb-core and kerb-util
(which is a transitive dependency frmo kerb-core). Depending on
kerb-simplekdc pulls a bunch of other unnecessary classes into the
hadoop-common classpath.
This changes the hadoop-common pom to depend only on kerb-core.
hadoop-minikdc already had the appropriate dependency on kerb-simplekdc
so it continues to pull in what it needs.
Signed-off-by: Todd Lipcon <todd@apache.org>
2019-04-10 08:49:46 -07:00
bb8dda2bf9
HADOOP-12890. Fix typo in AbstractService. Contributed by Gabor Liptak.
2019-04-08 15:26:12 +09:00
ab645b3caa
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang.
2019-04-08 15:18:45 +09:00
72f4b9cd68
HADOOP-15242. Fix typos in hadoop-functions.sh. Contributed by Ray Chiang.
2019-04-08 13:20:21 +09:00
0d47d283a6
HADOOP-10848. Cleanup calling of sun.security.krb5.Config.
2019-04-08 10:02:34 +09:00
c90736350b
HADOOP-16208. Do Not Log InterruptedException in Client.
...
Contributed by David Mollitor.
2019-04-04 16:15:57 +01:00
7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
e62cbcbc83
HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom implementations. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-03 14:29:52 -07:00
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
d7979079ea
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.
2019-04-03 12:59:39 -06:00
3b0c5016b2
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags
...
Signed-off-by: Todd Lipcon <todd@apache.org>
2019-04-03 10:56:33 -07:00
aaaf856f4b
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str
2019-04-03 13:16:59 +09:00
cf268114c9
HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-02 12:24:55 -07:00
61d19110d4
HADOOP-16218. Findbugs warning of null param to non-nullable method in Configuration with Guava update. ( #655 )
...
Change-Id: I461e518ce9a4730b91a8138ad55b39e9a4b0a4b8
2019-04-02 09:15:11 +01:00
ebd0d21538
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.
2019-04-02 10:51:47 +09:00
f41f938b2e
HADOOP-16199. KMSLoadBlanceClientProvider does not select token correctly. Contributed by Xiaoyu Yao.
...
This closes #642 .
2019-03-28 21:55:31 -07:00
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
...
Author: Gabor Bota
2019-03-28 15:59:25 +00:00
d18d0859eb
HADOOP-16181. HadoopExecutors shutdown Cleanup.
...
Author: David Mollitor <david.mollitor@cloudera.com>
2019-03-22 10:29:27 +00:00
246ab77f28
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
2019-03-22 10:26:24 +00:00
5f6e225166
YARN-9363. Replaced debug logging with SLF4J parameterized log message.
...
Contributed by Prabhu Joseph
2019-03-18 13:57:18 -04:00
5446e3cb8a
HADOOP-16167. Fixed Hadoop shell script for Ubuntu 18.
...
Contributed by Daniel Templeton
2019-03-18 13:04:49 -04:00
8c95cb9d6b
HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
2019-03-18 08:13:43 -07:00
2db38abffc
HDFS-14328. [Clean-up] Remove NULL check before instanceof in TestGSet
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I5b9f0e66664714d7c5bbfa30492a09f770626711
2019-03-18 15:10:26 +01:00
2064ca015d
YARN-9349. Changed logging to use slf4j api.
...
Contributed by Prabhu Joseph
2019-03-15 19:20:59 -04:00
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
f74159c8fc
HADOOP-16166. TestRawLocalFileSystemContract fails with build Docker container running on Mac.
...
Also provided similar fix for Windows.
2019-03-13 09:33:24 -07:00
2a54feabb2
HDFS-14347. [SBN Read] Restore a comment line mistakenly removed in ProtobufRpcEngine. Contributed by Fengnan Li.
2019-03-11 18:59:15 -07:00
Josh Rosen
Gopal V
Steve Loughran
Chen Liang
Inigo Goiri
Christopher Gregorian
Ajay Kumar
Ayush Saxena
Takanobu Asanuma
Surendra Singh Lilhore
Prabhu Joseph
Wei-Chiu Chuang
Shweta Yakkali
Adam Antal
Gabor Bota
cxorm
dineshchitlangia
Eric Yang
Wanqiang Ji
Erik Krogen
Huan-Ping Su
Akira Ajisaka
Sammi Chen
Sahil Takiar
Ben Roling
Alexis Daboville
David Mollitor
Bharat Viswanadham
Peter Bacsko
Siyao Meng
Vinayakumar B
Giovanni Matteo Fumarola
Sahil Takiar
Ben Roling
Sean Mackrory
Anu Engineer
Erik Krogen
Zsombor Gegesy
Kenneth Yang
Arpit Agarwal
Todd Lipcon
Steve Loughran
Lokesh Jain
Xiaoyu Yao
Matt Foley
Konstantin V Shvachko