aebac6d2d2
HADOOP-16461. Regression: FileSystem cache lock parses XML within the lock.
...
Contributed by Gopal V.
Change-Id: If6654f850e9c24ee0d9519a46fd6269b18e1a7a4
2019-07-26 11:32:13 +01:00
07530314c2
HADOOP-9844. NPE when trying to create an error message response of SASL RPC
...
This closes #55
Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-26 17:53:18 +09:00
a1251addff
HADOOP-16431. Remove useless log in IOUtils.java and ExceptionDiags.java.
...
This closes #1091
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-24 10:04:39 +09:00
4317d33232
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories.
...
Contributed by Steve Loughran and Gabor Bota.
This
* Asks S3Guard to determine the empty directory status.
* Has S3A's root directory rm("/") command to always return false (as abfs does)
* Documents that object stores MAY do this
* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.
It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".
Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108
2019-07-23 14:52:03 +01:00
eb36b09cb7
HADOOP-16443. Improve help text for setfacl --set option.
...
Contributed by S O'Donnell.
Change-Id: I1da46c4c414a5d2b07ee15867508f0799440a413
2019-07-23 10:24:07 +01:00
7f1b76ca35
HADOOP-13868. [s3a] New default for S3A multi-part configuration ( #1125 )
2019-07-19 09:49:59 -06:00
d545f9c290
HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise
...
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise
Contributed by: Josh Rosen
2019-07-18 23:19:38 +01:00
b4466a3b0a
HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
...
Contributed by Gopal V and Atilla Magyar.
Change-Id: I066d5eece332a1673594de0f9b484443f95530ec
2019-07-17 13:50:02 +01:00
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3.
...
Contributed by Steve Loughran
Contains
- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently
This patch doesn't fix the underlying problem but it
* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones
The outcome is that tests should pass consistently and if problems occur we have better diagnostics.
Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
2019-07-12 13:02:25 +01:00
8fb5ca3f40
HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.
2019-07-11 13:23:25 -07:00
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists.
...
Contributed by Steve Loughran.
This patch
* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.
Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
2019-07-11 18:15:34 +01:00
aa9f0850e8
HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena.
2019-07-02 13:22:30 -07:00
129576f628
HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian.
2019-06-24 12:09:17 -07:00
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar.
2019-06-24 11:59:07 -07:00
b52fd05d42
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma.
2019-06-24 22:03:04 +05:30
ba221dc50e
HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma.
2019-06-24 22:03:04 +05:30
559cb11551
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails.
2019-06-24 22:03:03 +05:30
f40e10b349
HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar.
2019-06-24 22:03:03 +05:30
371452e260
HADOOP-15989. Synchronized at CompositeService#removeService is not required. Contributed by Prabhu Joseph.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-21 17:05:13 -07:00
840d02ca5b
HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 19:57:48 -07:00
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.
...
Contributed by Steve Loughran.
Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
2019-06-20 09:56:40 +01:00
3ab77d9bc9
HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao.
2019-06-18 09:58:29 -07:00
dd4a7633ec
HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 23:47:57 -07:00
56c621c68f
HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 22:56:00 -07:00
10311c30b0
HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong.
2019-06-17 19:55:56 -07:00
1e92db5a1e
HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend.
2019-06-17 18:29:34 -07:00
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones).
...
Contributed by Gabor Bota.
Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
2019-06-16 17:05:01 +01:00
076618677d
HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm.
2019-06-14 20:05:20 -07:00
c7554ffd5c
HADOOP-16373. Fix typo in FileSystemShell#test documentation ( #968 )
2019-06-14 14:36:04 -07:00
54f9f75a44
HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles.
2019-06-14 10:17:25 +09:00
4ea6c2f457
HADOOP-16354. Enable AuthFilter as default for WebHDFS.
...
Contributed by Prabhu Joseph
2019-06-11 18:41:08 -04:00
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours`
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-11 18:40:35 +09:00
d6d95d2686
HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 10:23:18 -07:00
76b94c274f
HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen.
2019-06-07 14:20:44 -07:00
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables.
...
Contributed by Steve Loughran
Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
2019-06-07 18:26:10 +01:00
14552d19e3
HADOOP-16344. Make DurationInfo public unstable.
...
Second attempt: imports fixed up.
Contributed by Huan-Ping Su and Kai Xie.
Change-Id: Ib2a85dd4b2d12b840692cc854cc53ddb3bdde7d5
2019-06-07 12:55:17 +01:00
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java
2019-06-07 10:27:41 +09:00
1f244b4227
Revert "HADOOP-16344. Make DurationInfo public unstable."
...
This reverts commit 829848ba2e
2019-06-06 17:49:31 +01:00
829848ba2e
HADOOP-16344. Make DurationInfo public unstable.
...
Contributed by Huan-Ping Su
Change-Id: I64a94cf382f9db78b4ef49a3912f25f0d906af7c
2019-06-06 12:52:49 +01:00
294695dd57
HADOOP-16314. Make sure all web end points are covered by the same authentication filter.
...
Contributed by Prabhu Joseph
2019-06-05 18:55:13 -04:00
d1aad44490
HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.
2019-06-05 21:33:00 +08:00
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"
...
This reverts commit b067f8acaa
2019-06-05 13:54:55 +01:00
7724d8031b
Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors"
...
This reverts commit 5906268f0d
2019-06-05 13:54:42 +01:00
827a84778a
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer -follow-on patch.
...
This follow-on patch to HADOOP-16266 fixes up the problem where logs were being full of
stack traces because the timeout passed down to select was in nanos, whereas the API
expected millis.
Contributed by Erik Krogen.
Change-Id: I5c6e9ddf68127b1d7e0ca0e179d036eb9941e445
2019-06-04 15:56:47 +01:00
6f5a36c13c
HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:21:58 -07:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
93d7630062
HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-24 18:42:46 +02:00
f96a2df38d
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
2019-05-23 10:28:37 -07:00
ea0b1d8fba
HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation.
...
Contributed by Prabhu Joseph
2019-05-23 11:36:32 -04:00
a771e2a638
HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang.
2019-05-23 13:59:42 +09:00
5906268f0d
HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors
2019-05-21 11:30:45 -06:00
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.
...
Contributed by Ben Roling.
S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.
You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.
This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.
For a consistent experience, upgrade all clients to use the latest hadoop version.
2019-05-19 22:29:54 +01:00
4cb3da6ac7
HADOOP-16248. MutableQuantiles leak memory under heavy load.
...
Contributed by Alexis Daboville,
2019-05-17 15:15:22 +01:00
b067f8acaa
HADOOP-16050: s3a SSL connections should use OpenSSL
...
(cherry picked from commit aebf229c17
2019-05-16 08:57:54 -06:00
2713dcf6e9
HADOOP-16307. Intern User Name and Group Name in FileStatus.
...
Author: David Mollitor
2019-05-16 16:02:07 +02:00
d4c8858586
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy.
2019-05-15 17:41:36 -07:00
389e640f0c
HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao.
2019-05-13 11:46:16 -07:00
f257497b0f
HADOOP-16299. [JDK 11] Build fails without specifying -Djavac.version=11
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-09 14:49:46 +09:00
96dc5cedfe
HADOOP-16293. AuthenticationFilterInitializer doc has speudo instead of pseudo.
...
Author: Prabhu Joseph
2019-05-08 10:18:20 +01:00
713e8a27ae
HADOOP-16238. Add the possbility to set SO_REUSEADDR in IPC Server Listener. Contributed by Peter Bacsko.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-07 17:48:27 -07:00
93f2283a69
HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-06 15:47:33 -07:00
f1875b205e
HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena.
2019-05-03 11:22:14 +05:30
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
4877f0aa51
HDFS-3246: pRead equivalent for direct read path ( #597 )
...
HDFS-3246: pRead equivalent for direct read path
Contributed by Sahil Takiar
2019-04-30 14:52:16 -07:00
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.
2019-04-30 11:53:26 +01:00
a703dae25e
HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.
2019-04-24 10:39:00 -06:00
f4ab9370f5
HADOOP-16026:Replace incorrect use of system property user.name.
...
Contributed by Dinesh Chitlangia.
2019-04-22 14:02:13 -07:00
fb1c549139
HDFS-14374. Expose total number of delegation tokens in AbstractDelegationTokenSecretManager. Contributed by CR Hota.
2019-04-22 13:32:08 -07:00
1ddb48872f
HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin.
2019-04-22 08:16:57 -07:00
b1120d27ab
HADOOP-16249. Make CallerContext LimitedPrivate scope to Public.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-16 11:18:21 +09:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
626fec652b
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-04-12 17:37:51 -07:00
87407553ef
HADOOP-16243. Change Log Level to trace in NetUtils.java. Contributed by chencan.
2019-04-10 13:21:04 -07:00
813cee1a18
HDFS-14420. Fix typo in KeyShell console. Contributed by Hu Xiaodong.
2019-04-10 11:18:40 -07:00
65deb1ac42
HADOOP-16179. hadoop-common pom should not depend on kerb-simplekdc
...
The hadoop-common pom currently has a dependency on kerb-simplekdc. In
fact, the only classes used from Kerby are in kerb-core and kerb-util
(which is a transitive dependency frmo kerb-core). Depending on
kerb-simplekdc pulls a bunch of other unnecessary classes into the
hadoop-common classpath.
This changes the hadoop-common pom to depend only on kerb-core.
hadoop-minikdc already had the appropriate dependency on kerb-simplekdc
so it continues to pull in what it needs.
Signed-off-by: Todd Lipcon <todd@apache.org>
2019-04-10 08:49:46 -07:00
bb8dda2bf9
HADOOP-12890. Fix typo in AbstractService. Contributed by Gabor Liptak.
2019-04-08 15:26:12 +09:00
ab645b3caa
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang.
2019-04-08 15:18:45 +09:00
72f4b9cd68
HADOOP-15242. Fix typos in hadoop-functions.sh. Contributed by Ray Chiang.
2019-04-08 13:20:21 +09:00
c90736350b
HADOOP-16208. Do Not Log InterruptedException in Client.
...
Contributed by David Mollitor.
2019-04-04 16:15:57 +01:00
7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
e62cbcbc83
HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom implementations. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-03 14:29:52 -07:00
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
d7979079ea
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.
2019-04-03 12:59:39 -06:00
3b0c5016b2
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags
...
Signed-off-by: Todd Lipcon <todd@apache.org>
2019-04-03 10:56:33 -07:00
aaaf856f4b
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str
2019-04-03 13:16:59 +09:00
cf268114c9
HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-02 12:24:55 -07:00
61d19110d4
HADOOP-16218. Findbugs warning of null param to non-nullable method in Configuration with Guava update. ( #655 )
...
Change-Id: I461e518ce9a4730b91a8138ad55b39e9a4b0a4b8
2019-04-02 09:15:11 +01:00
ebd0d21538
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.
2019-04-02 10:51:47 +09:00
f41f938b2e
HADOOP-16199. KMSLoadBlanceClientProvider does not select token correctly. Contributed by Xiaoyu Yao.
...
This closes #642 .
2019-03-28 21:55:31 -07:00
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
...
Author: Gabor Bota
2019-03-28 15:59:25 +00:00
d18d0859eb
HADOOP-16181. HadoopExecutors shutdown Cleanup.
...
Author: David Mollitor <david.mollitor@cloudera.com>
2019-03-22 10:29:27 +00:00
246ab77f28
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
2019-03-22 10:26:24 +00:00
5f6e225166
YARN-9363. Replaced debug logging with SLF4J parameterized log message.
...
Contributed by Prabhu Joseph
2019-03-18 13:57:18 -04:00
5446e3cb8a
HADOOP-16167. Fixed Hadoop shell script for Ubuntu 18.
...
Contributed by Daniel Templeton
2019-03-18 13:04:49 -04:00
8c95cb9d6b
HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
2019-03-18 08:13:43 -07:00
2db38abffc
HDFS-14328. [Clean-up] Remove NULL check before instanceof in TestGSet
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I5b9f0e66664714d7c5bbfa30492a09f770626711
2019-03-18 15:10:26 +01:00
2064ca015d
YARN-9349. Changed logging to use slf4j api.
...
Contributed by Prabhu Joseph
2019-03-15 19:20:59 -04:00
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
f74159c8fc
HADOOP-16166. TestRawLocalFileSystemContract fails with build Docker container running on Mac.
...
Also provided similar fix for Windows.
2019-03-13 09:33:24 -07:00
2a54feabb2
HDFS-14347. [SBN Read] Restore a comment line mistakenly removed in ProtobufRpcEngine. Contributed by Fengnan Li.
2019-03-11 18:59:15 -07:00
4ad295a4f1
HDFS-14270.[SBN Read] Add trace level logging for stateId in RPC Server. Contributed by Shweta Yakkali.
2019-03-11 13:48:06 -07:00
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-09 16:00:34 +00:00
2b94e51a8f
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host.
...
Author: Praveen Krishna <praveenkrishna@tutanota.com>
2019-03-07 11:06:34 +00:00
a55fc36299
HDDS-1093. Configuration tab in OM/SCM ui is not displaying the correct values.
2019-03-06 17:43:57 -08:00
618e009ac0
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 15:04:06 -08:00
6192c1fe3b
Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar."
...
This reverts commit f5a4b43a49
2019-03-06 15:02:18 -08:00
f5a4b43a49
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 14:58:45 -08:00
3c5b7136e2
HADOOP-16150. Added concat method to ChecksumFS as unsupported operation.
...
Contributed by Steve Loughran
(cherry picked from commit 8b517e7ad6
2019-03-05 13:32:00 -05:00
686c0141ef
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately.
...
Contributed by Stephen O'Donnell.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 14:09:00 +00:00
e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-03-05 14:10:08 +09:00
fe7551f21b
HADOOP-16162. Remove unused Job Summary Appender configurations from log4j.properties
...
This closes #551
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-03-05 13:41:33 +09:00
9fcd89ab93
HADOOP-16148. Cleanup LineReader Unit Test.
...
Contributed by David Mollitor.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-04 23:08:12 +00:00
0d61facd37
HADOOP-15889. Add hadoop.token configuration parameter to load tokens. Contributed by Íñigo Goiri
2019-02-28 10:34:28 -08:00
feccd282fe
HADOOP-16107. Update ChecksumFileSystem createFile/openFile API to generate checksum.
...
Contributed by Steve Loughran
2019-02-27 15:53:41 -05:00
9192f71e21
HADOOP-16127. In ipc.Client, put a new connection could happen after stop.
2019-02-26 15:14:21 -08:00
52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util.
...
Contributed by Abhishek Modi
2019-02-26 17:10:41 +00:00
59ba3552d3
HDFS-14299. ViewFs: Correct error message for read only operations. Contributed by hu xiaodong.
2019-02-26 12:40:32 +05:30
a6ab37192a
HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko.
2019-02-25 14:35:02 -08:00
ba4e7bd192
HADOOP-16125. Support multiple bind users in LdapGroupsMapping. Contributed by Lukas Majercak.
2019-02-25 13:39:13 -08:00
0edb0c51dc
HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections.
2019-02-25 13:15:28 -08:00
f7a27cdee4
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.
2019-02-23 09:35:36 -08:00
a87e458432
HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-20 18:17:04 -08:00
371a6db59a
HDFS-14273. Fix checkstyle issues in BlockLocation's method javadoc
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I546aa4a0fe7f83b53735acd9925f366b2f1a00e2
2019-02-20 15:36:53 -08:00
dabfeab785
YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg.
2019-02-15 14:49:49 +09:00
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
fd026863b1
HDFS-14241. Provide feedback on successful renameSnapshot and deleteSnapshot. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-13 19:48:02 -08:00
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
00c5ffaee2
HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena.
2019-02-13 16:44:32 +05:30
7b11b404a3
HADOOP-16097. Provide proper documentation for FairCallQueue. Contributed by Erik Krogen.
2019-02-13 11:16:04 +08:00
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
...
(accidentally mixed in two patches)
This reverts commit fa8cd1bf28
2019-02-07 21:57:22 +00:00
308f3168fa
Make upstream aware of 3.1.2 release
...
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
2019-02-05 14:03:18 -08:00
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
7f46d13dac
HADOOP-16079. Token.toString faulting if any token listed can't load.
...
Contributed by Steve Loughran.
2019-02-01 14:31:47 +00:00
bcc3a79f58
HADOOP-16084. Fix the comment for getClass in Configuration. Contributed by Fengnan Li.
2019-01-31 10:06:05 -08:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
b3bc94ebfd
HDFS-14236. Lazy persist copy/ put fails with ViewFs.
2019-01-29 16:45:44 -08:00
d1714c20e9
Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh."
...
This reverts commit 1d523279da
2019-01-29 15:43:09 -08:00
1d523279da
HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 09:02:57 -08:00
0dd35e218f
HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user.
...
Contributed by He Xiaoqiao
2019-01-22 18:59:36 -05:00
a463cf75a0
HADOOP-15787. [JDK11] TestIPC.testRTEDuringConnectionSetup fails. Contributed by Zsolt Venczel.
2019-01-22 10:19:05 +09:00
2e2508b8e3
Make 3.2.0 aware to other branches
2019-01-21 21:24:51 +05:30
dacc1a759e
HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena.
2019-01-18 13:55:12 +05:30
01405df87f
HADOOP-15941. Addendum patch. Contributed by Takanobu Asanuma.
2019-01-16 10:28:08 +09:00
ccc43621ab
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration.
...
Contributed by luhuachao.
2019-01-15 13:15:18 +00:00
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
35fa3bd685
HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak.
2019-01-11 18:07:19 -08:00
bf08f4abae
HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian.
2019-01-11 14:01:23 -08:00
01cb958af4
HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen.
2019-01-11 12:51:07 -08:00
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
d4ca907da6
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper
2019-01-11 11:24:05 +09:00
33c009a45d
HADOOP-15909. KeyProvider class should implement Closeable. Contributed
...
by Kuhu Shukla.
2019-01-10 13:21:37 -06:00
c634589ab2
Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh."
...
This reverts commit ecdeaa7e6a
2019-01-09 17:24:58 -06:00
709ddb10e5
HADOOP-15941. [JDK 11] Compilation failure: package com.sun.jndi.ldap is not visible.
2019-01-09 13:55:25 +09:00
4ab5260b7e
HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
2019-01-08 17:04:57 -08:00
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
...
This reverts commit d7152332b3
2019-01-08 14:51:30 +09:00
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
d43af8b3db
HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop.
...
Contributed by Bolke de Bruin
2019-01-04 17:54:15 -05:00
f4e18242bd
HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke.
2019-01-04 09:56:58 -08:00
51427cbdfb
HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang.
2019-01-04 21:49:06 +05:30
ecdeaa7e6a
HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-03 09:56:07 -08:00
57198bc511
HADOOP-15962. The buffer size is small when unpacking tar archives. Contributed by BELUGA BEHR.
2018-12-28 15:38:47 +09:00
bd2a59e508
HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
2018-12-24 09:39:20 -08:00
fa8550337d
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
ef3e1929dd
HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
cae279610b
HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng.
2018-12-24 09:34:01 -08:00
b73fb70f97
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2018-12-24 09:34:01 -08:00
cbc2f932eb
HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
8c49135078
HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
a65bb97f5d
HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
d2e85b0b6d
HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
66ec3a67e8
HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
e27708c2da
HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov.
2018-12-24 09:34:00 -08:00
64b7cf59bd
HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
151c8ddbe4
HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
2018-12-24 09:33:59 -08:00
e880660a20
HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
091ad974cd
HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
9bf0696c73
HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
c9d73437e8
HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
feb2664ac4
HDFS-13946. Log longest FSN write/read lock held stack trace.
2018-12-22 23:09:59 +08:00
763e96ea2b
HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan.
2018-12-22 12:22:24 +05:30
3961690037
HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne
2018-12-20 09:29:48 -06:00
d963575918
HDFS-13661. Ls command with e option fails when the filesystem is not HDFS.
2018-12-20 10:03:33 +09:00
788e7473a4
HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena.
2018-12-17 11:24:57 +05:30
1ea29b7385
HADOOP-16000. Remove TLSv1 and SSLv2Hello from the default value of hadoop.ssl.enabled.protocols
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-12-15 10:53:52 +09:00
76efeacd5f
HADOOP-15995. Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping. Contributed by Lukas Majercak.
2018-12-12 11:10:24 -08:00
202926ac33
HADOOP-15808. Harden Token service loader use.
...
Contributed by Steve Loughran.
2018-12-11 17:33:49 +00:00
aa89492f29
Revert "HADOOP-15852. Refactor QuotaUsage. Contributed by Beluga Behr."
...
This reverts commit fb9deed41d
2018-12-04 12:57:28 -08:00
db2d8b01c6
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset.
...
Contributed by Steve Loughran
2018-12-04 15:35:43 +00:00
fb9deed41d
HADOOP-15852. Refactor QuotaUsage. Contributed by Beluga Behr.
2018-12-03 13:25:02 -08:00
c9a3aa64dc
HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak.
2018-12-03 12:10:05 -08:00
38ea3814bd
HADOOP-15922. Revert patch 004.
2018-11-30 12:49:46 -05:00
c9bfca217f
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin.
2018-11-30 13:13:23 +09:00
d0edd37269
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
...
This reverts commit 829a2e4d27
2018-11-29 17:52:11 +00:00
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
2018-11-29 15:12:17 +00:00
a9d96948ed
HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter.
...
Contributed by He Xiaoqiao
2018-11-26 13:49:19 -05:00
23b441c225
HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena.
2018-11-26 18:11:01 +05:30
6934a65402
HADOOP-15358. SFTPConnectionPool connections leakage. Contributed by Mikhail Pryakhin.
2018-11-23 09:48:26 +01:00
1a00b4e325
Add 2.9.2 release notes and changes documents.
2018-11-20 14:25:18 +09:00
460a94a10f
HDFS-14045. Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN. Contributed by Jiandan Yang.
2018-11-15 10:58:57 -08:00
75291e6d53
HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore
2018-11-13 20:22:58 +05:30
3c9d97b8f7
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds.
...
Contributed by LiXin Ge.
2018-11-12 12:58:05 +00:00
89b49167a5
HADOOP-15903. Allow HttpServer2 to discover resources in /static when symlinks are used. Contributed by Inigo Goiri.
2018-11-08 14:52:24 -08:00
0b6625a973
MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan
2018-11-07 08:20:49 -06:00
ab6aa4c726
YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
2018-11-06 08:40:59 -06:00
cb8d679c95
HADOOP-15687. Credentials class should allow access to aliases.
...
Author: Lars Francke <lars.francke@gmail.com>
2018-11-03 16:21:29 +00:00
44e37b4fd9
HADOOP-15885. Add base64 (urlString) support to DTUtil. Contributed by Inigo Goiri.
2018-11-02 10:54:12 -07:00
62d98ca92a
HADOOP-15855. Review hadoop credential doc, including object store details.
...
Contributed by Steve Loughran.
2018-10-30 15:58:04 +00:00
63e7134df6
Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao."
...
This reverts commit fb2b72e6fc
2018-10-28 17:04:59 -07:00
bfb9adc2b9
HADOOP-9567. Provide auto-renewal for keytab based logins. Contributed by Hrishikesh Gadre, Gary Helmling and Harsh J.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-27 08:59:47 -07:00
fb2b72e6fc
HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao.
2018-10-25 09:33:31 -07:00
8f97d6f2cd
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe.
2018-10-24 13:49:28 +09:00
823bb5dda8
MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter)
2018-10-23 15:28:13 -07:00
635786a511
HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang.
2018-10-23 14:53:45 -07:00
dd268a64d3
Revert "HADOOP-15836. Review of AccessControlList. Contributed by BELUGA BEHR."
...
This reverts commit 00254d7b8c
2018-10-23 09:23:03 -07:00
52cb766ad0
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility.
...
Contributed by Wei-Chiu Chuang.
2018-10-22 11:21:12 +01:00
e2a9fa8448
HADOOP-15821. Move YARN Registry to Hadoop Registry.
...
Contributed by Íñigo Goiri
2018-10-19 19:46:48 -04:00
00254d7b8c
HADOOP-15836. Review of AccessControlList. Contributed by BELUGA BEHR.
2018-10-19 13:57:43 -07:00
d93d515af5
HDDS-651. Rename o3 to o3fs for Filesystem.
2018-10-17 14:19:17 -07:00
9abda83947
HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe
2018-10-17 14:38:42 -05:00
24dc068a36
HADOOP-11100. Support to configure ftpClient.setControlKeepAliveTimeout.
...
Contributed by Adam Antal.
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-17 11:34:50 -07:00
41b3603b5b
HADOOP-15861. Move DelegationTokenIssuer to the right path.
...
Contributed by Wei-Chiu Chuang.
2018-10-17 11:01:53 +01:00
c2288ac45b
YARN-8448. AM HTTPS Support for AM communication with RMWeb proxy. (Contributed by Robert Kanter)
2018-10-16 13:36:26 -07:00
7fe1a40a6b
HADOOP-15853. TestConfigurationDeprecation leaves behind a temp file, resulting in a license issue (ayushtkn via rkanter)
2018-10-15 15:02:37 -07:00
b6fc72a025
HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
2018-10-15 10:51:55 -07:00
3c1fe073d2
HDDS-445. Create a logger to print out all of the incoming requests.
...
Contributed by Bharat Viswanadham.
2018-10-12 16:27:54 -07:00
5ec86b445c
HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
...
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
2018-10-12 09:35:52 -07:00
d91d47bc73
HADOOP-15717. TGT renewal thread does not log IOException (snemeth via rkanter)
2018-10-11 15:35:44 -07:00
64f2b32d57
HADOOP-15676. Cleanup TestSSLHttpServer. Contributed by Szilard Nemeth.
2018-10-11 15:12:36 -07:00
7b57f2f71f
HADOOP-15785. [JDK10] Javadoc build fails on JDK 10 in hadoop-common. Contributed by Dinesh Chitlangia.
2018-10-11 13:51:51 +09:00
f261c31937
HADOOP-15708. Reading values from Configuration before adding deprecations make it impossible to read value with deprecated key (zsiegl via rkanter)
2018-10-10 18:51:37 -07:00
cdc4350718
HADOOP-15839. Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys.
...
Contributed by Steve Loughran.
2018-10-10 17:00:37 +01:00
6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter.
2018-10-10 10:16:57 +09:00
bf04f19456
YARN-8845. Removed unused hadoop.registry.rm reference.
...
Contributed by Íñigo Goiri
2018-10-09 15:54:45 -04:00
9bbeb52486
HADOOP-15775. [JDK9] Add missing javax.activation-api dependency. Contributed by Akira Ajisaka.
2018-10-09 13:06:14 +09:00
f13e231025
HADOOP-15820. ZStandardDecompressor native code sets an integer field as a long. Contributed by Jason Lowe
2018-10-05 09:06:02 -05:00
046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota
2018-10-02 21:22:49 -07:00
31fa9bb799
HADOOP-15767. [JDK10] Building native package on JDK10 fails due to missing javah. Contributed by Takanobu Asanuma.
2018-10-03 11:29:18 +09:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
6fab6886f6
HDFS-13936. Multipart upload to HDFS to support 0 byte upload. Contributed by Ewan Higgs.
2018-10-02 14:05:35 +02:00
5c8d90763c
HADOOP-15764. Addendum patch: Fix NPE in SecurityUtil.
2018-09-28 10:03:22 +09:00
c62b440421
HADOOP-15693. Typo in Secure Mode documentation. Contributed by Thomas Levine.
2018-09-26 10:30:43 +09:00
93b0f540ed
HADOOP-15783. [JDK10] TestSFTPFileSystem.testGetModifyTime fails.
2018-09-25 16:03:28 +09:00
d060cbea48
HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran.
2018-09-24 21:19:47 +05:30
0def61482b
Merge branch 'HADOOP-15407' into trunk
2018-09-22 21:19:12 -06:00
429a07e08c
HADOOP-15764. [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement. Contributed by Akira Ajisaka.
2018-09-20 15:13:55 +02:00
3da94a36e2
HADOOP-15756. [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement. Contributed by Akira Ajisaka.
2018-09-20 14:53:21 +02:00
646874c326
HADOOP-15748. S3 listing inconsistency can raise NPE in globber.
...
Contributed by Steve Loughran.
2018-09-20 13:04:52 +01:00
7ad27e97f0
HADOOP-15736. Trash : Negative Value For Deletion Interval Leads To Abnormal Behaviour. Contributed by Ayush Saxena.
2018-09-20 09:31:35 +05:30
a30b4f9e71
HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen.
2018-09-19 13:22:37 -07:00
e71f61ecb8
HADOOP-15755. StringUtils#createStartupShutdownMessage throws NPE when args is null. Contributed by Lokesh Jain and Dinesh Chitlangia
2018-09-18 15:57:33 -05:00
1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch.
...
Contributed by Steve Loughran.
2018-09-18 10:56:56 +01:00
ee051ef9fe
HADOOP-15742. Log if ipc backoff is enabled in CallQueueManager. Contributed by Ryan Wu.
2018-09-18 11:10:33 +08:00
9475fd902a
HADOOP-15745. ABFS: Add ABFS configuration to ConfigRedactor.
2018-09-17 19:54:01 +00:00
81dc4a995c
HADOOP-15663. ABFS: Simplify configuration.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
ce03a93f78
HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests.
...
Contributed by Steve Loughran and Da Zhou.
2018-09-17 19:54:01 +00:00
f044deedbb
HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.
...
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
2018-09-17 19:54:01 +00:00
ef5c776a42
HADOOP-15733. Correct the log when Invalid emptier Interval configured. Contributed by Ayush Saxena
2018-09-14 07:32:27 +05:30
291dcf2247
HDDS-423. Introduce an ozone specific log4j.properties. Contributed by Elek, Marton.
2018-09-13 16:37:44 -07:00
5d084d7eca
HADOOP-12760. sun.misc.Cleaner has moved to a new location in OpenJDK 9. Contributed by Akira Ajisaka.
2018-09-13 20:42:09 +09:00
d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran)
2018-09-12 21:04:49 -07:00
c44088ac19
YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan.
2018-09-12 16:01:01 +05:30
2475fb0a1e
YARN-6856. [YARN-3409] Support CLI for Node Attributes Mapping. Contributed by Naganarasimha G R.
2018-09-12 16:00:59 +05:30
87f63b6479
HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran
2018-09-11 17:18:42 -07:00
4b8c2b1c1c
HADOOP-14671. Upgrade Apache Yetus to 0.8.0
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-09-11 08:55:34 -07:00
527288ef89
HDFS-13836. RBF: Handle mount table znode with null value. Contributed by yanghuafeng.
2018-09-06 16:47:54 -07:00
962089ab59
HADOOP-15657 Registering MutableQuantiles via Metric annotation. Contributed by Sushil Ks
2018-09-05 20:53:21 -07:00
9af96d4ed4
HADOOP-15707. Add IsActiveServlet to be used for Load Balancers. Contributed by Lukas Majercak.
2018-09-05 10:50:25 -07:00
e780556ae9
HADOOP-15696. KMS performance regression due to too many open file descriptors after Jetty migration. Contributed by Wei-Chiu Chuang.
2018-09-05 09:52:35 -07:00
9e96ac666d
HADOOP-10219. ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdowns.
...
Contributed by Kihwal Lee and Lukas Majercak.
2018-09-04 16:46:12 +01:00
f2c2a68ec2
HADOOP-15706. Typo in compatibility doc: SHOUD -> SHOULD
...
(Contributed by Laszlo Kollar via Daniel Templeton)
Change-Id: I6e2459d0700df7f3bad4eac8297a11690191c3ba
2018-08-30 09:12:36 -07:00
2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length.
...
Contributed by Ewan Higgs
2018-08-30 14:33:16 +01:00
d53a10b0a5
HADOOP-15705. Typo in the definition of "stable" in the interface classification
...
Change-Id: I3eae2143400a534903db4f186400561fc8d2bd56
2018-08-29 13:59:32 -07:00
f9c6fd9471
HADOOP-15633. fs.TrashPolicyDefault: Can't create trash directory. Contributed by Fei Hui.
2018-08-27 09:22:59 -07:00
a4121c71c2
HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko.
2018-08-24 18:27:30 -07:00
bed8cb6979
HADOOP-14314. The OpenSolaris taxonomy link is dead in InterfaceClassification.md. Contributed by Rui Gao.
2018-08-23 21:54:38 +09:00
34577d2c21
HADOOP-15679. ShutdownHookManager shutdown time needs to be configurable & extended.
...
Contributed by Steve Loughran.
2018-08-20 18:36:24 -07:00
79c97f6a0b
HADOOP-14624. Add GenericTestUtils.DelayAnswer that accept slf4j logger API. Contributed by Ian Pickering and Wenxin He.
2018-08-17 14:40:00 -07:00
60ffec9f79
HADOOP-9214. Create a new touch command to allow modifying atime and mtime. Contributed by Hrishikesh Gadre.
2018-08-17 11:18:09 -07:00
a17eed1b87
HDFS-13822. speedup libhdfs++ build (enable parallel build). Contributed by Allen Wittenauer and Pradeep Ambati
2018-08-17 12:25:36 -05:00
8d7c93186e
HADOOP-15674. Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite. Contributed by Szilard Nemeth.
2018-08-17 10:09:23 -07:00
77b015000a
HADOOP-8807. Update README and website to reflect HADOOP-8662. Contributed by Andras Bokor.
2018-08-17 11:10:29 +02:00
edeb2a356a
HADOOP-15655. Enhance KMS client retry behavior. Contributed by Kitti Nanasi.
2018-08-16 22:42:03 -07:00
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering.
2018-08-16 00:31:59 +09:00
7e822ec246
HDDS-324. Addendum: remove the q letter which is accidentally added to the hadoop-functions.sh.
2018-08-14 20:54:33 +02:00
b4031a8f1b
HDDS-324. Use pipeline name as Ratis groupID to allow datanode to report pipeline info. Contributed by Mukul Kumar Singh.
2018-08-13 12:47:49 -07:00
da9a39eed1
HADOOP-15583. Stabilize S3A Assumed Role support.
...
Contributed by Steve Loughran.
2018-08-08 22:57:24 -07:00
49c687608b
Make 3.1.1 awared by other branches
2018-08-08 13:02:12 -07:00
9499df7b81
HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi.
2018-08-08 10:40:20 -07:00
2ec97abb2e
HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran.
2018-08-08 13:50:23 +02:00
6ed8593d18
HDDS-301. ozone command shell does not contain subcommand to run ozoneFS commands. Contributed by Nilotpal Nandi.
2018-08-07 16:09:53 +05:30
2e4e02b4df
HDFS-13792. Fix FSN read/write lock metrics name. Contributed by Chao Sun.
2018-08-07 09:32:51 +08:00
f2e29acbfa
HADOOP-15476. fix logging for split-dns multihome . Contributed by Ajay Kumar.
2018-08-01 12:32:01 -07:00
e8f952ef06
HADOOP-15637. LocalFs#listLocatedStatus does not filter out hidden .crc files. Contributed by Erik Krogen.
2018-07-30 10:25:07 -07:00
77721f39e2
HADOOP-15593. Fixed NPE in UGI spawnAutoRenewalThreadForUserCreds.
...
Contributed by Gabor Bota
2018-07-26 18:35:36 -04:00
be150a17b1
HDFS-13622. mkdir should print the parent directory in the error message when parent directories do not exist. Contributed by Shweta.
2018-07-26 10:24:32 -07:00
9089790cab
HADOOP-15611. Log more details for FairCallQueue. Contributed by Ryan Wu.
2018-07-26 18:08:28 +08:00
5f0b924360
HADOOP-15395. DefaultImpersonationProvider fails to parse proxy user config if username has . in it. Contributed by Ajay Kumar.
2018-07-25 21:09:11 +05:30
81d59506e5
HADOOP-15609. Retry KMS calls when SSLHandshakeException occurs. Contributed by Kitti Nanasi.
2018-07-24 21:45:43 -07:00
849c45db18
HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
...
(Contributed by BELUGA BEHR via Daniel Templeton)
Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
2018-07-24 16:05:27 -07:00
6bec03cfc8
HADOOP-15612. Improve exception when tfile fails to load LzoCodec. (gera)
2018-07-24 14:32:30 -07:00
8461278833
YARN-8544. [DS] AM registration fails when hadoop authorization is enabled. Contributed by Bibin A Chundatt.
2018-07-24 13:09:17 +05:30
17e26163ec
HADOOP-15586. Fix wrong log statement in AbstractService. (Szilard Nemeth via Haibo Chen)
2018-07-23 11:18:25 -07:00
993ec026d1
HADOOP-15596. Stack trace should not be printed out when running hadoop key commands. Contributed by Kitti Nanasi.
2018-07-20 19:46:46 -07:00
e6873dfde0
YARN-7300. DiskValidator is not used in LocalDirAllocator. (Szilard Nemeth via Haibo Chen)
2018-07-19 16:27:11 -07:00
ccf2db7fc2
HADOOP-15614. TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably fails. Contributed by Weiwei Yang.
2018-07-19 11:13:37 -05:00
d215357718
HDFS-13485. DataNode WebHDFS endpoint throws NPE. Contributed by Siyao Meng.
2018-07-16 15:45:55 -07:00
d2874e0417
HDFS-13690. Improve error message when creating encryption zone while KMS is unreachable. Contributed by Kitti Nanasi.
2018-07-16 13:19:53 -07:00
0c7a578927
HADOOP-15598. DataChecksum calculate checksum is contented on hashtable synchronization. Contributed by Prasanth Jayachandran.
2018-07-16 11:32:45 -07:00
88625f5cd9
HADOOP-15531. Use commons-text instead of commons-lang in some classes to fix deprecation warnings. Contributed by Takanobu Asanuma.
2018-07-13 11:42:12 -04:00
4f3f9391b0
HADOOP-15316. GenericTestUtils can exceed maxSleepTime. Contributed by Adam Antal.
2018-07-12 17:24:01 +02:00
162228e8db
HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng.
2018-07-11 10:02:08 -07:00
4a08ddfa68
HADOOP-15568. fix some typos in the .sh comments. Contributed by Steve Loughran.
2018-07-09 15:43:38 -04:00
def9d94a40
HADOOP-15591. KMSClientProvider should log KMS DT acquisition at INFO level. Contributed by Kitti Nanasi.
2018-07-09 12:01:52 -07:00
7a68ac607c
HDFS-13674. Improve documentation on Metrics. Contributed by Chao Sun.
2018-07-09 14:27:34 +08:00
061b168529
HDDS-167. Rename KeySpaceManager to OzoneManager. Contributed by Arpit Agarwal.
2018-07-06 12:09:05 -07:00
498e3bfb6b
HADOOP-15571. Multiple FileContexts created with the same configuration object should be allowed to have different umask. Contributed by Vinod Kumar Vavilapalli.
2018-07-06 11:56:09 -07:00
f51da9c4d1
HADOOP-15554. Improve JIT performance for Configuration parsing. Contributed by Todd Lipcon.
2018-07-02 18:31:21 +02:00
d36f6b9e93
HADOOP-15548: Randomize local dirs. Contributed by Jim Brennan.
2018-06-29 18:18:32 +00:00
384764cdea
YARN-8409. Fixed NPE in ActiveStandbyElectorBasedElectorService.
...
Contributed by Chandni Singh
2018-06-28 12:39:49 -04:00
d3fa83a44b
HDFS-13705:The native ISA-L library loading failure should be made warning rather than an error message. Contributed by Shashikant Banerjee.
2018-06-28 09:21:56 -07:00
ddbff7c8d3
HADOOP-14313. Replace/improve Hadoop's byte[] comparator. Contributed by Vikas Vishwakarma.
2018-06-28 14:58:40 +09:00
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.
2018-06-28 14:37:22 +09:00
7a3c6e9c3c
HADOOP-15550. Avoid static initialization of ObjectMappers
2018-06-25 15:36:45 -07:00
abc3e4bad9
HADOOP-15458. TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows. Contributed by Xiao Liang.
2018-06-25 09:50:27 -07:00
1ba4e62304
HADOOP-14396. Add builder interface to FileContext.
...
Contributed by Lei (Eddy) Xu.
2018-06-25 14:38:57 +01:00
6432128622
YARN-8444: NodeResourceMonitor crashes on bad swapFree value. Contributed by Jim Brennan.
2018-06-22 17:15:29 +00:00
59de967954
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption
2018-06-21 10:32:52 -07:00
43541a1890
HADOOP-15551. Avoid use of Arrays.stream in Configuration.addTags
2018-06-20 16:43:10 -07:00
32f867a6a9
HDFS-13682. Cannot create encryption zone after KMS auth token expires.
2018-06-20 15:58:26 -07:00
2d87592fc6
HADOOP-15527. Improve delay check for stopping processes.
...
Fixed script location.
Contributed by Vinod Kumar Vavilapalli
2018-06-19 13:38:13 -04:00
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
3905fdb793
HADOOP-15523. Shell command timeout given is in seconds whereas it is taken as millisec while scheduling. Contributed by Bilwa S T.
2018-06-17 12:12:01 +05:30
8e7548d33b
HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.
2018-06-13 20:05:55 +08:00
108da85320
HADOOP-15527. Improve delay check for stopping processes.
...
Contributed by Vinod Kumar Vavilapalli
2018-06-12 20:40:32 -04:00
aeaf9fec62
HADOOP-15532. TestBasicDiskValidator fails with NoSuchFileException. Contributed by Giovanni Matteo Fumarola.
2018-06-12 14:16:14 -07:00
2df73dace0
YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri.
2018-06-12 17:13:09 +05:30
7c3dc39083
Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release.
...
(cherry picked from commit 5a82f10e32
2018-06-11 15:13:18 -07:00
18201b882a
HADOOP-15499. Performance severe drops when running RawErasureCoderBenchmark with NativeRSRawErasureCoder. Contributed by Sammi Chen.
2018-06-11 13:53:37 +08:00
ef0118b91e
HADOOP-15520. Add tests for various org.apache.hadoop.util classes.
...
Contributed by Arash Nabili
2018-06-09 15:33:38 +01:00
12be8bad7d
HADOOP-15516. Add test cases to cover FileUtil#readLink. Contributed by Giovanni Matteo Fumarola.
2018-06-07 13:34:52 -07:00
f494f0b896
HADOOP-15512. Clean up Shell from JDK7 workarounds.
...
Contributed by Zsolt Venczel.
2018-06-07 15:55:49 +01:00
db81f85e84
HADOOP-15513. Add additional test cases to cover some corner cases for FileUtil#symlink. Contributed by Giovanni Matteo Fumarola.
2018-06-06 11:44:17 -07:00
ba4011d64f
HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.
2018-06-05 21:18:31 -07:00
1b0d4f4606
HDFS-13547. Add ingress port based sasl resolver. Contributed by Chen Liang.
2018-06-05 11:51:29 -07:00
6d5e87aec2
HADOOP-15507. Add MapReduce counters about EC bytes read.
2018-06-04 21:13:47 -07:00
04cf699dd5
YARN-8388. TestCGroupElasticMemoryController.testNormalExit() hangs on Linux. (Miklos Szegedi via Haibo Chen)
2018-06-04 14:23:08 -07:00
4880d890ee
YARN-8375. TestCGroupElasticMemoryController fails surefire build. (Miklos Szegedi via Haibo Chen)
2018-06-01 16:29:13 -07:00
Gopal V
Steve Loughran
sunlisheng
Steve Loughran
S O'Donnell
Sean Mackrory
Josh Rosen
Chen Liang
Inigo Goiri
Christopher Gregorian
Ajay Kumar
Ayush Saxena
Takanobu Asanuma
Surendra Singh Lilhore
Prabhu Joseph
Wei-Chiu Chuang
Shweta Yakkali
Adam Antal
Gabor Bota
cxorm
dineshchitlangia
Eric Yang
Wanqiang Ji
Erik Krogen
Huan-Ping Su
Akira Ajisaka
Sammi Chen
Sahil Takiar
Ben Roling
Alexis Daboville
David Mollitor
Bharat Viswanadham
Peter Bacsko
Siyao Meng
Vinayakumar B
Giovanni Matteo Fumarola
Sahil Takiar
Ben Roling
Sean Mackrory
Anu Engineer
Erik Krogen
Kenneth Yang
Arpit Agarwal
Todd Lipcon
Lokesh Jain
Xiaoyu Yao
Matt Foley
Konstantin V Shvachko
Praveen Krishna
Vivek Ratnavel Subramanian
Sahil Takiar
coder-chenzhi
Tsz Wo Nicholas Sze
Abhishek Modi
Yongjun Zhang
Daryn Sharp
Yiqun Lin
Wangda Tan
Da Zhou
Hanisha Koneru
Pranay Singh
Sunil G
luhuachao
Kihwal Lee
Jason Lowe
Chao Sun
Plamen Jeliazkov
Ewan Higgs
Márton Elek
Lars Francke
Hrishikesh Gadre
Haibo Chen
Jitendra Pandey
Robert Kanter
Xiao Chen
Aaron Fabbri
Ewan Higgs
Thomas Marquardt
Brahma Reddy Battula
bibinchundatt
Naganarasimha
Mingliang Liu
Allen Wittenauer
Vrushali C
Daniel Templeton
John Zhuge
Giovanni Matteo Fumarola
Mukul Kumar Singh
Gera Shegalov
Andrew Wang
Eric E Payne
Chris Douglas
Rohith Sharma K S
Sammi Chen