Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HBASE-24136 Add release branch report to git-jira-release-audit tool 

* enhancements to git_jira_release_audit.py
  * add aforementioned release branch report
  * include default values in help doc output
  * swap default db to a file on disk instead of in memory
  * set logger to match file name
  * add separate sql query log at DEBUG level
* more detailed usage info in README.md, including example audit query
* update entries in fallback_actions.csv

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
This commit is contained in:
Nick Dimiduk 2020-04-08 16:04:40 -07:00 committed by Nick Dimiduk
parent 1fb693b998
commit b4af0a0943
4 changed files with 245 additions and 89 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
dev-support/git-jira-release-audit/.gitignore vendored
View File

@ -2,3 +2,4 @@
*.log *.log
*.svg *.svg
venv venv
new_for_*.csv

253
dev-support/git-jira-release-audit/README.md
View File

@ -43,7 +43,7 @@ $ ./venv/bin/pip install -r ./requirements.txt
Successfully installed... Successfully installed...
``` ```
## Usage ## Basic Usage
The tool provides basic help docs. The tool provides basic help docs.
@ -54,6 +54,7 @@ usage: git_jira_release_audit.py [-h] [--populate-from-git POPULATE_FROM_GIT]
[--db-path DB_PATH] [--db-path DB_PATH]
[--initialize-db INITIALIZE_DB] [--initialize-db INITIALIZE_DB]
[--report-new-for-release-line REPORT_NEW_FOR_RELEASE_LINE] [--report-new-for-release-line REPORT_NEW_FOR_RELEASE_LINE]
[--report-new-for-release-branch REPORT_NEW_FOR_RELEASE_BRANCH]
[--git-repo-path GIT_REPO_PATH] [--git-repo-path GIT_REPO_PATH]
[--remote-name REMOTE_NAME] [--remote-name REMOTE_NAME]
[--development-branch DEVELOPMENT_BRANCH] [--development-branch DEVELOPMENT_BRANCH]
@ -61,7 +62,9 @@ usage: git_jira_release_audit.py [-h] [--populate-from-git POPULATE_FROM_GIT]
[--release-line-regexp RELEASE_LINE_REGEXP] [--release-line-regexp RELEASE_LINE_REGEXP]
[--parse-release-tags PARSE_RELEASE_TAGS] [--parse-release-tags PARSE_RELEASE_TAGS]
[--fallback-actions-path FALLBACK_ACTIONS_PATH] [--fallback-actions-path FALLBACK_ACTIONS_PATH]
[--jira-url JIRA_URL] [--jira-url JIRA_URL] --branch-1-fix-version
BRANCH_1_FIX_VERSION --branch-2-fix-version
BRANCH_2_FIX_VERSION
optional arguments: optional arguments:
-h, --help show this help message and exit -h, --help show this help message and exit
@ -69,90 +72,169 @@ optional arguments:
Building the audit database: Building the audit database:
--populate-from-git POPULATE_FROM_GIT --populate-from-git POPULATE_FROM_GIT
When true, populate the audit database from the Git When true, populate the audit database from the Git
repository. repository. (default: True)
--populate-from-jira POPULATE_FROM_JIRA --populate-from-jira POPULATE_FROM_JIRA
When true, populate the audit database from Jira. When true, populate the audit database from Jira.
(default: True)
--db-path DB_PATH Path to the database file, or leave unspecified for a --db-path DB_PATH Path to the database file, or leave unspecified for a
transient db. transient db. (default: audit.db)
--initialize-db INITIALIZE_DB --initialize-db INITIALIZE_DB
When true, initialize the database tables. This is When true, initialize the database tables. This is
destructive to the contents of an existing database. destructive to the contents of an existing database.
(default: False)
Generating reports: Generating reports:
--report-new-for-release-line REPORT_NEW_FOR_RELEASE_LINE --report-new-for-release-line REPORT_NEW_FOR_RELEASE_LINE
Builds a report of the Jira issues that are new on the Builds a report of the Jira issues that are new on the
target release line, not present on any of the target release line, not present on any of the
associated release branches. (i.e., on branch-2 but associated release branches. (i.e., on branch-2 but
not branch-{2.0,2.1,...}) not branch-{2.0,2.1,...}) (default: None)
--report-new-for-release-branch REPORT_NEW_FOR_RELEASE_BRANCH
Builds a report of the Jira issues that are new on the
target release branch, not present on any of the
previous release branches. (i.e., on branch-2.3 but
not branch-{2.0,2.1,...}) (default: None)
Interactions with the Git repo: Interactions with the Git repo:
--git-repo-path GIT_REPO_PATH --git-repo-path GIT_REPO_PATH
Path to the git repo, or leave unspecified to infer Path to the git repo, or leave unspecified to infer
from the current file's path. from the current file's path. (default:
./git_jira_release_audit.py)
--remote-name REMOTE_NAME --remote-name REMOTE_NAME
The name of the git remote to use when identifying The name of the git remote to use when identifying
branches. Default: 'origin' branches. Default: 'origin' (default: origin)
--development-branch DEVELOPMENT_BRANCH --development-branch DEVELOPMENT_BRANCH
The name of the branch from which all release lines The name of the branch from which all release lines
originate. Default: 'master' originate. Default: 'master' (default: master)
--development-branch-fix-version DEVELOPMENT_BRANCH_FIX_VERSION --development-branch-fix-version DEVELOPMENT_BRANCH_FIX_VERSION
The Jira fixVersion used to indicate an issue is The Jira fixVersion used to indicate an issue is
committed to the development branch. Default: '3.0.0' committed to the development branch. (default: 3.0.0)
--release-line-regexp RELEASE_LINE_REGEXP --release-line-regexp RELEASE_LINE_REGEXP
A regexp used to identify release lines. A regexp used to identify release lines. (default:
branch-\d+$)
--parse-release-tags PARSE_RELEASE_TAGS --parse-release-tags PARSE_RELEASE_TAGS
When true, look for release tags and annotate commits When true, look for release tags and annotate commits
according to their release version. An Expensive according to their release version. An Expensive
calculation, disabled by default. calculation, disabled by default. (default: False)
--fallback-actions-path FALLBACK_ACTIONS_PATH --fallback-actions-path FALLBACK_ACTIONS_PATH
Path to a file containing _DB.Actions applicable to Path to a file containing _DB.Actions applicable to
specific git shas. specific git shas. (default: fallback_actions.csv)
--branch-1-fix-version BRANCH_1_FIX_VERSION --branch-1-fix-version BRANCH_1_FIX_VERSION
The Jira fixVersion used to indicate an issue is The Jira fixVersion used to indicate an issue is
committed to the specified release line branch committed to the specified release line branch
(default: None)
--branch-2-fix-version BRANCH_2_FIX_VERSION --branch-2-fix-version BRANCH_2_FIX_VERSION
The Jira fixVersion used to indicate an issue is The Jira fixVersion used to indicate an issue is
committed to the specified release line branch committed to the specified release line branch
(default: None)
Interactions with Jira: Interactions with Jira:
--jira-url JIRA_URL A URL locating the target JIRA instance. --jira-url JIRA_URL A URL locating the target JIRA instance. (default:
https://issues.apache.org/jira)
``` ```
### Build a Database
This invocation will build a "simple" database, correlating commits to
branches. It omits gathering the detailed release tag data, so it runs pretty
quickly.
Example Run: Example Run:
```shell script ```shell script
$ ./venv/bin/python3 ./git_jira_release_audit.py \ $ ./venv/bin/python3 ./git_jira_release_audit.py \
--db-path=audit.db \ --db-path=audit.db \
--remote-name=apache-rw \
--development-branch-fix-version=3.0.0 \ --development-branch-fix-version=3.0.0 \
--branch-1-fix-version=1.5.0 \ --branch-1-fix-version=1.7.0 \
--branch-2-fix-version=2.3.0 --branch-2-fix-version=2.4.0
INFO:root:apache-rw/branch-1 has 4046 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb. INFO:git_jira_release_audit.py:origin/branch-1.0 has 1433 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
INFO:root:apache-rw/branch-1.0 has 1433 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb. INFO:git_jira_release_audit.py:origin/branch-1.1 has 2111 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
INFO:root:apache-rw/branch-1.1 has 2111 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb. INFO:git_jira_release_audit.py:origin/branch-1.2 has 2738 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
INFO:root:apache-rw/branch-1.2 has 2738 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb. INFO:git_jira_release_audit.py:origin/branch-1.3 has 3296 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
INFO:root:apache-rw/branch-1.3 has 3287 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb. INFO:git_jira_release_audit.py:origin/branch-1.4 has 3926 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
INFO:root:apache-rw/branch-1.4 has 3912 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb. INFO:git_jira_release_audit.py:origin/branch-2 has 3325 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
INFO:root:apache-rw/branch-2 has 3080 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60. INFO:git_jira_release_audit.py:origin/branch-2.0 has 2198 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
INFO:root:apache-rw/branch-2.0 has 2194 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60. INFO:git_jira_release_audit.py:origin/branch-2.1 has 2749 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
INFO:root:apache-rw/branch-2.1 has 2705 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60. INFO:git_jira_release_audit.py:origin/branch-2.2 has 2991 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
INFO:root:apache-rw/branch-2.2 has 2927 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60. INFO:git_jira_release_audit.py:origin/branch-2.3 has 3312 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
INFO:root:retrieving 5653 jira_ids from the issue tracker INFO:git_jira_release_audit.py:retrieving 5850 jira_ids from the issue tracker
apache-rw/branch-1 100%|██████████████████████████████████████████████████████| 4046/4046 [08:23<00:00, 8.04 commit/s] origin/branch-1 100%|████████████████████████████████████| 4084/4084 [00:00<00:00, 9805.33 commit/s]
apache-rw/branch-1.0 100%|████████████████████████████████████████████████████| 1433/1433 [03:49<00:00, 6.26 commit/s] origin/branch-1.0 100%|█████████████████████████████████| 1433/1433 [00:00<00:00, 10479.89 commit/s]
apache-rw/branch-1.1 100%|████████████████████████████████████████████████████| 2111/2111 [05:16<00:00, 6.68 commit/s] origin/branch-1.1 100%|█████████████████████████████████| 2111/2111 [00:00<00:00, 10280.60 commit/s]
apache-rw/branch-1.2 100%|████████████████████████████████████████████████████| 2738/2738 [06:26<00:00, 7.10 commit/s] origin/branch-1.2 100%|██████████████████████████████████| 2738/2738 [00:00<00:00, 8833.51 commit/s]
apache-rw/branch-1.3 100%|████████████████████████████████████████████████████| 3287/3287 [07:21<00:00, 7.46 commit/s] origin/branch-1.3 100%|██████████████████████████████████| 3296/3296 [00:00<00:00, 9746.93 commit/s]
apache-rw/branch-1.4 100%|████████████████████████████████████████████████████| 3912/3912 [08:08<00:00, 8.02 commit/s] origin/branch-1.4 100%|██████████████████████████████████| 3926/3926 [00:00<00:00, 9750.96 commit/s]
apache-rw/branch-2 100%|█████████████████████████████████████████████████████| 3080/3080 [03:29<00:00, 14.74 commit/s] origin/branch-2 100%|████████████████████████████████████| 3325/3325 [00:00<00:00, 9688.14 commit/s]
apache-rw/branch-2.0 100%|████████████████████████████████████████████████████| 2194/2194 [04:56<00:00, 7.42 commit/s] origin/branch-2.0 100%|██████████████████████████████████| 2198/2198 [00:00<00:00, 8804.18 commit/s]
apache-rw/branch-2.1 100%|███████████████████████████████████████████████████| 2705/2705 [03:17<00:00, 13.75 commit/s] origin/branch-2.1 100%|██████████████████████████████████| 2749/2749 [00:00<00:00, 9328.67 commit/s]
apache-rw/branch-2.2 100%|███████████████████████████████████████████████████| 2927/2927 [03:28<00:00, 14.09 commit/s] origin/branch-2.2 100%|██████████████████████████████████| 2991/2991 [00:00<00:00, 9215.56 commit/s]
fetch from Jira 100%|█████████████████████████████████████████████████████████| 5653/5653 [00:58<00:00, 98.29 issue/s] origin/branch-2.3 100%|██████████████████████████████████| 3312/3312 [00:00<00:00, 9063.19 commit/s]
fetch from Jira 100%|████████████████████████████████████████| 5850/5850 [10:40<00:00, 9.14 issue/s]
``` ```
With a populated database, query with sqlite: Optionally, the database can be build to include release tags, by specifying
`--parse-release-tags=true`. This is more time-consuming, but is necessary for
auditing discrepancies between git and Jira. Running the same command but
including this flag looks like this:
```shell script
origin/branch-1 100%|███████████████████████████████████████| 4084/4084 [08:58<00:00, 7.59 commit/s]
origin/branch-1.0 100%|█████████████████████████████████████| 1433/1433 [03:54<00:00, 6.13 commit/s]
origin/branch-1.1 100%|█████████████████████████████████████| 2111/2111 [41:26<00:00, 0.85 commit/s]
origin/branch-1.2 100%|█████████████████████████████████████| 2738/2738 [07:10<00:00, 6.37 commit/s]
origin/branch-1.3 100%|██████████████████████████████████| 3296/3296 [2h 33:13<00:00, 0.36 commit/s]
origin/branch-1.4 100%|██████████████████████████████████| 3926/3926 [7h 22:41<00:00, 0.15 commit/s]
origin/branch-2 100%|████████████████████████████████████| 3325/3325 [2h 05:43<00:00, 0.44 commit/s]
origin/branch-2.0 100%|█████████████████████████████████████| 2198/2198 [52:18<00:00, 0.70 commit/s]
origin/branch-2.1 100%|█████████████████████████████████████| 2749/2749 [17:09<00:00, 2.67 commit/s]
origin/branch-2.2 100%|█████████████████████████████████████| 2991/2991 [52:15<00:00, 0.95 commit/s]
origin/branch-2.3 100%|████████████████████████████████████| 3312/3312 [05:08<00:00, 10.74 commit/s]
fetch from Jira 100%|████████████████████████████████████████| 5850/5850 [10:46<00:00, 9.06 issue/s]
```
### Run a Report
With a database populated with branch information, the build-in reports can be
run.
`--report-new-for-release-line`
> Builds a report of the Jira issues that are new on the target release line,
not present on any of the associated release branches. (i.e., on branch-2 but
not branch-{2.0,2.1,...})
`--report-new-for-release-branch`
> Builds a report of the Jira issues that are new on the target release branch,
not present on any of the previous release branches. (i.e., on branch-2.3 but
not branch-{2.0,2.1,...})
Either way, the output is a csv file containing a summary of each JIRA id found
matching the report criteria.
Example Run:
```shell script
$ ./venv/bin/python3.7 ./git_jira_release_audit.py \
--populate-from-git=false \
--populate-from-jira=false \
--branch-1-fix-version=1.7.0 \
--branch-2-fix-version=2.4.0 \
--report-new-for-release-branch=origin/branch-2.3
INFO:git_jira_release_audit.py:retrieving 292 jira_ids from the issue tracker
INFO:git_jira_release_audit.py:generated report at new_for_origin-branch-2.3.csv
fetch from Jira 100%|████████████████████████████████████████| 292/292 [00:03<00:00, 114.01 issue/s]
$ head -n5 new_for_origin-branch-2.3.csv
key,issue_type,priority,summary,resolution,components
HBASE-21070,Bug,Critical,SnapshotFileCache won't update for snapshots stored in S3,Fixed,['snapshots']
HBASE-21773,Bug,Critical,rowcounter utility should respond to pleas for help,Fixed,['tooling']
HBASE-21505,Bug,Major,Several inconsistencies on information reported for Replication Sources by hbase shell status 'replication' command.,Fixed,['Replication']
HBASE-22057,Bug,Major,Impose upper-bound on size of ZK ops sent in a single multi(),Fixed,[]
```
### Explore the Database
With a populated database, query it with sqlite:
```shell script ```shell script
$ sqlite3 audit.db $ sqlite3 audit.db
@ -160,56 +242,71 @@ SQLite version 3.24.0 2018-06-04 14:10:15
Enter ".help" for usage hints. Enter ".help" for usage hints.
sqlite> -- count the number of distinct commits on a release branch sqlite> -- count the number of distinct commits on a release branch
sqlite> select count(distinct jira_id), branch from git_commits group by branch; sqlite> select count(distinct jira_id), branch from git_commits group by branch;
3406|apache-rw/branch-1 3437|origin/branch-1
1189|apache-rw/branch-1.0 1189|origin/branch-1.0
1728|apache-rw/branch-1.1 1728|origin/branch-1.1
2289|apache-rw/branch-1.2 2289|origin/branch-1.2
2779|apache-rw/branch-1.3 2788|origin/branch-1.3
3277|apache-rw/branch-1.4 3289|origin/branch-1.4
2666|apache-rw/branch-2 2846|origin/branch-2
1809|apache-rw/branch-2.0 1813|origin/branch-2.0
2289|apache-rw/branch-2.1 2327|origin/branch-2.1
2511|apache-rw/branch-2.2 2566|origin/branch-2.2
2839|origin/branch-2.3
sqlite> -- count the number of issues that will be in 2.3.0 that have not been released on any earlier
sqlite> -- version.
sqlite> select count(1) from (
select distinct jira_id from git_commits where branch = 'apache-rw/branch-2' except
select distinct jira_id from git_commits where branch in
('apache-rw/branch-2.0', 'apache-rw/branch-2.1', 'apache-rw/branch-2.2'));
169
sqlite> -- find the issues for which the git commit record and JIRA fixVersion disagree sqlite> -- find the issues for which the git commit record and JIRA fixVersion disagree
sqlite> -- this query requires the database be built with --parse-release-tags
sqlite> select g.jira_id, g.git_tag, j.fix_version sqlite> select g.jira_id, g.git_tag, j.fix_version
from git_commits g from git_commits g
inner join jira_versions j inner join jira_versions j
on g.jira_id = j.jira_id on g.jira_id = j.jira_id
and g.branch = 'apache-rw/branch-2.2' and g.branch = 'origin/branch-2.2'
and g.git_tag is not null and g.git_tag is not null
and j.fix_version like '2.2.%' and j.fix_version like '2.2.%'
and g.git_tag != j.fix_version; and g.git_tag != j.fix_version;
HBASE-22941|2.2.2|2.2.1 HBASE-22941|2.2.2|2.2.1
sqlite> -- show jira non-1.x fixVersions for all issues on branch-2 but not on any sqlite> -- show jira fixVersions for all issues on branch-2.3 but not on any earlier
sqlite> -- branch-2.x release branch; i.e., issues that are missing a fixVersion or sqlite> -- branch; i.e., issues that are missing a fixVersion or are marked for
sqlite> -- are marked for a release other than (3.0.0, 2.3.0) sqlite> -- a release other than the expected (3.0.0, 2.3.0).
sqlite> select g.jira_id, j.fix_version sqlite> -- this query requires the database be built with --parse-release-tags
from ( sqlite> select jira_id, fix_version
select distinct jira_id from git_commits where branch = 'apache-rw/branch-2' except FROM jira_versions
select distinct jira_id from git_commits where branch in WHERE jira_id in (
(select distinct branch from git_commits where branch like 'apache-rw/branch-2.%')) g SELECT distinct jira_id
left join jira_versions j FROM git_commits
on g.jira_id = j.jira_id WHERE branch = 'origin/branch-2.3'
and j.fix_version not like '1.%' EXCEPT SELECT distinct jira_id
where ( FROM git_commits
j.fix_version is null WHERE branch IN (
OR j.fix_version not in ('3.0.0', '2.3.0')) SELECT distinct branch
order by g.jira_id desc; FROM git_commits
HBASE-23683|2.2.4 WHERE branch != 'origin/branch-2.3'))
AND fix_version NOT IN ('3.0.0', '2.3.0')
ORDER BY jira_id;
HBASE-22321|1.5.0
HBASE-22360|2.2.0
HBASE-22405|2.2.0
HBASE-22555|2.4.0
HBASE-23032|connector-1.0.1 HBASE-23032|connector-1.0.1
HBASE-23032|hbase-filesystem-1.0.0-alpha2 HBASE-23032|hbase-filesystem-1.0.0-alpha2
HBASE-22405|2.2.0 HBASE-23604|HBASE-18095
HBASE-22360|2.2.0 HBASE-23633|2.4.0
HBASE-22321| HBASE-23647|HBASE-18095
HBASE-22283|2.2.0 HBASE-23648|HBASE-18095
HBASE-23731|HBASE-18095
HBASE-23741|2.4.0
HBASE-23752|HBASE-18095
HBASE-23804|HBASE-18095
HBASE-23851|master
HBASE-23936|2.4.0
HBASE-23937|2.4.0
HBASE-23977|2.4.0
HBASE-24002|2.4.0
HBASE-24033|2.4.0
HBASE-24037|2.4.0
HBASE-24073|master
HBASE-24075|2.4.0
HBASE-24080|2.4.0
HBASE-24080|master
``` ```

10
dev-support/git-jira-release-audit/fallback_actions.csv
View File

@ -27,8 +27,10 @@ hexsha,action,jira_id
05f8e94191ef6a63baadf56d6114d7d0317796f2,SKIP, 05f8e94191ef6a63baadf56d6114d7d0317796f2,SKIP,
0791b878422eadf00b55076338f09bf059f39f0c,SKIP, 0791b878422eadf00b55076338f09bf059f39f0c,SKIP,
07f9f3d38cf4d0d01044ab28d90a50a1a009f6b8,SKIP, 07f9f3d38cf4d0d01044ab28d90a50a1a009f6b8,SKIP,
0bff1305134b9c3a0bcad21900f5af68a8aedb4a,SKIP,
10f00547627076d79d77cf58dd2deaece2287084,ADD,HBASE-22330 10f00547627076d79d77cf58dd2deaece2287084,ADD,HBASE-22330
10f3b77748a02a2c11635c33964929c0474e890d,SKIP, 10f3b77748a02a2c11635c33964929c0474e890d,SKIP,
1196e42362312080d3c523c107b5e8fefef9e57e,SKIP,
1404d5a97331ecc63db53971f5cb7329cb40ce67,ADD,HBASE-15203 1404d5a97331ecc63db53971f5cb7329cb40ce67,ADD,HBASE-15203
14a869828fe481697d29b2d6e4135e8026039a38,SKIP, 14a869828fe481697d29b2d6e4135e8026039a38,SKIP,
1546613e76b1013a08ebc179c2c22bfeb44f3a4a,SKIP, 1546613e76b1013a08ebc179c2c22bfeb44f3a4a,SKIP,
@ -42,6 +44,7 @@ hexsha,action,jira_id
1b3557649c9ee682c7f135ca52a0e3cd10cb9219,SKIP, 1b3557649c9ee682c7f135ca52a0e3cd10cb9219,SKIP,
1c46250bef9ef9be9c255d61bda69ff7792ed551,SKIP, 1c46250bef9ef9be9c255d61bda69ff7792ed551,SKIP,
1cb7d0e82ad64f37fbd6de950b74081b0d5eddf3,SKIP, 1cb7d0e82ad64f37fbd6de950b74081b0d5eddf3,SKIP,
1d988afc9d2065a51fe74d0553f0943ef540dfaa,SKIP,
1eaef185327171b3dd3edb303e08cfe85186e745,SKIP, 1eaef185327171b3dd3edb303e08cfe85186e745,SKIP,
1eb8ac6fe9dd0c15cdb52f66ced4136316c06465,SKIP, 1eb8ac6fe9dd0c15cdb52f66ced4136316c06465,SKIP,
2068804d7510e8c1f822b5db3cd4585455f6e7e7,SKIP, 2068804d7510e8c1f822b5db3cd4585455f6e7e7,SKIP,
@ -52,6 +55,7 @@ hexsha,action,jira_id
259d12f7397679c6b0d0a4788e5a37f65fd49f20,SKIP, 259d12f7397679c6b0d0a4788e5a37f65fd49f20,SKIP,
267bce0590c39570ddb935921e34bda35e3aa44c,SKIP, 267bce0590c39570ddb935921e34bda35e3aa44c,SKIP,
278828333c44493ccbaa7db26a788b2756632034,SKIP, 278828333c44493ccbaa7db26a788b2756632034,SKIP,
27c1f2f978142b7bb4135e53c3ae067c5608c9fb,SKIP,
288794d68ba5bd4d1fd8d5c315cee972019dcb3d,ADD,HBASE-22330 288794d68ba5bd4d1fd8d5c315cee972019dcb3d,ADD,HBASE-22330
28f07451a5dddf0ab3988b32b8672654fdbc5b58,SKIP, 28f07451a5dddf0ab3988b32b8672654fdbc5b58,SKIP,
2ba542d74c2d9e78332c8c94289d1295752d8072,SKIP, 2ba542d74c2d9e78332c8c94289d1295752d8072,SKIP,
@ -97,7 +101,9 @@ hexsha,action,jira_id
4eb84651a2b6d02d2074143308cef5d0f4b856a3,SKIP, 4eb84651a2b6d02d2074143308cef5d0f4b856a3,SKIP,
4f5b22bc19cb8d24ced5d42ebd9794cfd83bae85,SKIP, 4f5b22bc19cb8d24ced5d42ebd9794cfd83bae85,SKIP,
54337870eda5649ab7bb81ed01c9dd25d59204f2,SKIP, 54337870eda5649ab7bb81ed01c9dd25d59204f2,SKIP,
558ee079fd04dfab8e61eca10ee98ab5bac89dfa,SKIP,
58ab201be341f02829286f036a7401d0806eb999,SKIP, 58ab201be341f02829286f036a7401d0806eb999,SKIP,
58b63e04c4af99a5730efb0a7e553be4d950e6a5,SKIP,
5a16c15d7f51087a50511a2e0730f547c97a033f,SKIP, 5a16c15d7f51087a50511a2e0730f547c97a033f,SKIP,
5b5ff1d8b2cc43f78acaf9bc960be382dc6c34f7,SKIP, 5b5ff1d8b2cc43f78acaf9bc960be382dc6c34f7,SKIP,
5fa15dd7488433ea610ff5e92161409d20565690,SKIP, 5fa15dd7488433ea610ff5e92161409d20565690,SKIP,
@ -170,6 +176,7 @@ b3d55441b8174c704ada4585603f6bcfca298843,SKIP,
b65231d04dbc565a578ce928e809aa51f5439857,SKIP, b65231d04dbc565a578ce928e809aa51f5439857,SKIP,
b6549007b313e8f3aa993d5c1ebd29c84ccb7b7b,SKIP, b6549007b313e8f3aa993d5c1ebd29c84ccb7b7b,SKIP,
b6d4fc955fe0fc41f5225f1cc2e3e4b92029251c,SKIP, b6d4fc955fe0fc41f5225f1cc2e3e4b92029251c,SKIP,
b9c676cdc048c52f927cfa906fd18ff412e4ca20,SKIP,
b9f5c6b065ebd572193c1fdc9d38557320b42fe6,SKIP, b9f5c6b065ebd572193c1fdc9d38557320b42fe6,SKIP,
bcadcef21048e4764f7ae8dec3ce52884f20c02c,SKIP, bcadcef21048e4764f7ae8dec3ce52884f20c02c,SKIP,
bcdc56ac76e4a26e53faa8301a441e94ee8614d7,SKIP, bcdc56ac76e4a26e53faa8301a441e94ee8614d7,SKIP,
@ -178,9 +185,11 @@ bd4e14db07ea32a45c3ef734e06d195a405da67c,SKIP,
bd4eba2b53b7af738fd9584511d737c4393d0855,SKIP, bd4eba2b53b7af738fd9584511d737c4393d0855,SKIP,
bef0616ef33306afca3060b96c2cba5f9762035d,SKIP, bef0616ef33306afca3060b96c2cba5f9762035d,SKIP,
c100fb835a54be6002fe9704349e726f27b15b7a,SKIP, c100fb835a54be6002fe9704349e726f27b15b7a,SKIP,
c5e0a1397b3c6a14612e4c5b66f995c02de4310b,SKIP,
c71da858ada94e1b93065f0b7caf3558942bc4da,SKIP, c71da858ada94e1b93065f0b7caf3558942bc4da,SKIP,
c89cfd3406823cf05fa83464c5ddee16bf0d473f,ADD,HBASE-17248 c89cfd3406823cf05fa83464c5ddee16bf0d473f,ADD,HBASE-17248
c89cfd3406823cf05fa83464c5ddee16bf0d473f,ADD,HBASE-17248 c89cfd3406823cf05fa83464c5ddee16bf0d473f,ADD,HBASE-17248
c8c2a875056f27c9af81293a504d75634cbc1fa5,SKIP,
c97905a962b88a0c68ca8a51c2e507daec81ca6d,SKIP, c97905a962b88a0c68ca8a51c2e507daec81ca6d,SKIP,
c9f506a2973e0acbd0d2df7b9353c9291f6c94a8,SKIP, c9f506a2973e0acbd0d2df7b9353c9291f6c94a8,SKIP,
cbb2c7e00d0c0b3f641250d981b9c87286d31058,ADD,HBASE-23069 cbb2c7e00d0c0b3f641250d981b9c87286d31058,ADD,HBASE-23069
@ -209,6 +218,7 @@ e40fcee6b54712b76d702af6937c3320c60df2b9,SKIP,
e501fe1a296be8fec0890e7e15414683aa3d933b,SKIP, e501fe1a296be8fec0890e7e15414683aa3d933b,SKIP,
e5349d589c000e395e12340e003aa9e2153afea6,SKIP, e5349d589c000e395e12340e003aa9e2153afea6,SKIP,
e5fb8214b2bfd6396539a4e8b6cf5f3cc5e9c06f,REVERT,HBASE-21874 e5fb8214b2bfd6396539a4e8b6cf5f3cc5e9c06f,REVERT,HBASE-21874
e869a20123afe326e198d35d110f5c0360ea244f,SKIP,
e8e45ef8f2fb91a870399636b492d5cee58a4c39,SKIP, e8e45ef8f2fb91a870399636b492d5cee58a4c39,SKIP,
e92a147e1961366e36a39577816994566e1e21c5,SKIP, e92a147e1961366e36a39577816994566e1e21c5,SKIP,
eacf3cb29641af1a68978d9bd7654f643a3aa3a1,SKIP, eacf3cb29641af1a68978d9bd7654f643a3aa3a1,SKIP,

Can't render this file because it contains an unexpected character in line 7 and column 3.

70
dev-support/git-jira-release-audit/git_jira_release_audit.py
View File

@ -30,12 +30,16 @@ import pathlib
import re import re
import sqlite3 import sqlite3
import time import time
import os
import enlighten import enlighten
import git import git
import jira import jira
LOG = logging.getLogger(os.path.basename(__file__))
class _DB: class _DB:
"""Manages an instance of Sqlite on behalf of the application. """Manages an instance of Sqlite on behalf of the application.
@ -46,6 +50,9 @@ class _DB:
Attributes: Attributes:
conn (:obj:`sqlite3.db2api.Connection`): The underlying connection object. conn (:obj:`sqlite3.db2api.Connection`): The underlying connection object.
""" """
SQL_LOG = LOG.getChild("sql")
class Action(enum.Enum): class Action(enum.Enum):
"""Describes an action to be taken against the database.""" """Describes an action to be taken against the database."""
ADD = 'ADD' ADD = 'ADD'
@ -54,6 +61,7 @@ class _DB:
def __init__(self, db_path, initialize_db, **_kwargs): def __init__(self, db_path, initialize_db, **_kwargs):
self._conn = sqlite3.connect(db_path) self._conn = sqlite3.connect(db_path)
self._conn.set_trace_callback(_DB.log_query)
if initialize_db: if initialize_db:
for table in 'git_commits', 'jira_versions': for table in 'git_commits', 'jira_versions':
@ -81,6 +89,10 @@ class _DB:
def __exit__(self, exc_type, exc_val, exc_tb): def __exit__(self, exc_type, exc_val, exc_tb):
self._conn.close() self._conn.close()
@staticmethod
def log_query(query):
_DB.SQL_LOG.debug(re.sub(r'\s+', ' ', query).strip())
@property @property
def conn(self): def conn(self):
""":obj:`sqlite3.db2api.Connection`: Underlying database handle.""" """:obj:`sqlite3.db2api.Connection`: Underlying database handle."""
@ -324,7 +336,7 @@ class _RepoReader:
def _resolve_ambiguity(self, commit): def _resolve_ambiguity(self, commit):
if commit.hexsha not in self._fallback_actions: if commit.hexsha not in self._fallback_actions:
logging.warning('Unable to resolve action for %s: %s', commit.hexsha, commit.summary) LOG.warning('Unable to resolve action for %s: %s', commit.hexsha, commit.summary)
return _DB.Action.SKIP, None return _DB.Action.SKIP, None
action, jira_id = self._fallback_actions[commit.hexsha] action, jira_id = self._fallback_actions[commit.hexsha]
if not jira_id: if not jira_id:
@ -354,7 +366,7 @@ class _RepoReader:
global MANAGER global MANAGER
commits = list(self._repo.iter_commits( commits = list(self._repo.iter_commits(
"%s...%s" % (origin_commit.hexsha, release_branch), reverse=True)) "%s...%s" % (origin_commit.hexsha, release_branch), reverse=True))
logging.info("%s has %d commits since its origin at %s.", release_branch, len(commits), LOG.info("%s has %d commits since its origin at %s.", release_branch, len(commits),
origin_commit) origin_commit)
counter = MANAGER.counter(total=len(commits), desc=release_branch, unit='commit') counter = MANAGER.counter(total=len(commits), desc=release_branch, unit='commit')
commits_since_release = list() commits_since_release = list()
@ -404,7 +416,7 @@ class _JiraReader:
database.""" database."""
global MANAGER global MANAGER
jira_ids = self._db.unique_jira_ids_from_git() jira_ids = self._db.unique_jira_ids_from_git()
logging.info("retrieving %s jira_ids from the issue tracker", len(jira_ids)) LOG.info("retrieving %s jira_ids from the issue tracker", len(jira_ids))
counter = MANAGER.counter(total=len(jira_ids), desc='fetch from Jira', unit='issue') counter = MANAGER.counter(total=len(jira_ids), desc='fetch from Jira', unit='issue')
chunk_size = 50 chunk_size = 50
chunks = [jira_ids[i:i + chunk_size] for i in range(0, len(jira_ids), chunk_size)] chunks = [jira_ids[i:i + chunk_size] for i in range(0, len(jira_ids), chunk_size)]
@ -429,7 +441,7 @@ class _JiraReader:
def fetch_issues(self, jira_ids): def fetch_issues(self, jira_ids):
"""Retrieve the specified jira Ids.""" """Retrieve the specified jira Ids."""
global MANAGER global MANAGER
logging.info("retrieving %s jira_ids from the issue tracker", len(jira_ids)) LOG.info("retrieving %s jira_ids from the issue tracker", len(jira_ids))
counter = MANAGER.counter(total=len(jira_ids), desc='fetch from Jira', unit='issue') counter = MANAGER.counter(total=len(jira_ids), desc='fetch from Jira', unit='issue')
chunk_size = 50 chunk_size = 50
chunks = [jira_ids[i:i + chunk_size] for i in range(0, len(jira_ids), chunk_size)] chunks = [jira_ids[i:i + chunk_size] for i in range(0, len(jira_ids), chunk_size)]
@ -501,17 +513,17 @@ class Auditor:
writer.writeheader() writer.writeheader()
for issue in issues: for issue in issues:
writer.writerow(issue) writer.writerow(issue)
logging.info('generated report at %s', filename) LOG.info('generated report at %s', filename)
def report_new_for_release_line(self, release_line): def report_new_for_release_line(self, release_line):
"""Builds a report of the Jira issues that are new on the target release line, not present """Builds a report of the Jira issues that are new on the target release line, not present
on any of the associated release branches. (i.e., on branch-2 but not on any of the associated release branches. (i.e., on branch-2 but not
branch-{2.0,2.1,...})""" branch-{2.0,2.1,...})"""
matches = [x for x in self._repo_reader.release_line_refs matches = [x for x in self._repo_reader.release_line_refs
if x.name == release_line or x.name.endswith('/%s' % release_line)] if x.name == release_line or x.remote_head == release_line]
release_line_ref = next(iter(matches), None) release_line_ref = next(iter(matches), None)
if not release_line_ref: if not release_line_ref:
logging.error('release line %s not found. available options are %s.', LOG.error('release line %s not found. available options are %s.',
release_line, [x.name for x in self._repo_reader.release_line_refs]) release_line, [x.name for x in self._repo_reader.release_line_refs])
return return
cursor = self._db.conn.execute(""" cursor = self._db.conn.execute("""
@ -525,6 +537,31 @@ class Auditor:
filename = 'new_for_%s.csv' % release_line.replace('/', '-') filename = 'new_for_%s.csv' % release_line.replace('/', '-')
Auditor._write_report(filename, issues) Auditor._write_report(filename, issues)
def report_new_for_release_branch(self, release_branch):
"""Builds a report of the Jira issues that are new on the target release branch, not present
on any of the previous release branches. (i.e., on branch-2.3 but not
branch-{2.0,2.1,...})"""
matches = [x for x in self._repo_reader.release_branch_refs
if x.name == release_branch or x.remote_head == release_branch]
release_branch_ref = next(iter(matches), None)
if not release_branch_ref:
LOG.error('release branch %s not found. available options are %s.',
release_branch, [x.name for x in self._repo_reader.release_branch_refs])
return
previous_branches = [x.name for x in self._repo_reader.release_branch_refs
if x.remote_head != release_branch_ref.remote_head]
query = (
"SELECT distinct jira_id FROM git_commits"
" WHERE branch = ?"
" EXCEPT SELECT distinct jira_id FROM git_commits"
f" WHERE branch IN ({','.join('?' for _ in previous_branches)})"
)
cursor = self._db.conn.execute(query, tuple([release_branch_ref.name] + previous_branches))
jira_ids = [x[0] for x in cursor.fetchall()]
issues = self._jira_reader.fetch_issues(jira_ids)
filename = 'new_for_%s.csv' % release_branch.replace('/', '-')
Auditor._write_report(filename, issues)
@staticmethod @staticmethod
def _str_to_bool(val): def _str_to_bool(val):
if not val: if not val:
@ -548,7 +585,7 @@ class Auditor:
building_group.add_argument( building_group.add_argument(
'--db-path', '--db-path',
help='Path to the database file, or leave unspecified for a transient db.', help='Path to the database file, or leave unspecified for a transient db.',
default=':memory:') default='audit.db')
building_group.add_argument( building_group.add_argument(
'--initialize-db', '--initialize-db',
help='When true, initialize the database tables. This is destructive to the contents' help='When true, initialize the database tables. This is destructive to the contents'
@ -561,6 +598,11 @@ class Auditor:
help=Auditor.report_new_for_release_line.__doc__, help=Auditor.report_new_for_release_line.__doc__,
type=str, type=str,
default=None) default=None)
report_group.add_argument(
'--report-new-for-release-branch',
help=Auditor.report_new_for_release_branch.__doc__,
type=str,
default=None)
git_repo_group = parser.add_argument_group('Interactions with the Git repo') git_repo_group = parser.add_argument_group('Interactions with the Git repo')
git_repo_group.add_argument( git_repo_group.add_argument(
'--git-repo-path', '--git-repo-path',
@ -580,7 +622,7 @@ class Auditor:
git_repo_group.add_argument( git_repo_group.add_argument(
'--development-branch-fix-version', '--development-branch-fix-version',
help='The Jira fixVersion used to indicate an issue is committed to the development' help='The Jira fixVersion used to indicate an issue is committed to the development'
+ ' branch. Default: \'3.0.0\'', + ' branch.',
default='3.0.0') default='3.0.0')
git_repo_group.add_argument( git_repo_group.add_argument(
'--release-line-regexp', '--release-line-regexp',
@ -612,7 +654,10 @@ class Auditor:
help='The Jira fixVersion used to indicate an issue is committed to the specified ' help='The Jira fixVersion used to indicate an issue is committed to the specified '
+ 'release line branch', + 'release line branch',
required=True) required=True)
return argparse.ArgumentParser(parents=[parent_parser]) return argparse.ArgumentParser(
parents=[parent_parser],
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
MANAGER = None MANAGER = None
@ -621,11 +666,11 @@ MANAGER = None
def main(): def main():
global MANAGER global MANAGER
logging.basicConfig(level=logging.INFO)
first_pass_parser, git_repo_group = Auditor._build_first_pass_parser() first_pass_parser, git_repo_group = Auditor._build_first_pass_parser()
first_pass_args, extras = first_pass_parser.parse_known_args() first_pass_args, extras = first_pass_parser.parse_known_args()
first_pass_args_dict = vars(first_pass_args) first_pass_args_dict = vars(first_pass_args)
with _DB(**first_pass_args_dict) as db: with _DB(**first_pass_args_dict) as db:
logging.basicConfig(level=logging.INFO)
repo_reader = _RepoReader(db, **first_pass_args_dict) repo_reader = _RepoReader(db, **first_pass_args_dict)
jira_reader = _JiraReader(db, **first_pass_args_dict) jira_reader = _JiraReader(db, **first_pass_args_dict)
second_pass_parser = Auditor._build_second_pass_parser( second_pass_parser = Auditor._build_second_pass_parser(
@ -641,6 +686,9 @@ def main():
if second_pass_args.report_new_for_release_line: if second_pass_args.report_new_for_release_line:
release_line = second_pass_args.report_new_for_release_line release_line = second_pass_args.report_new_for_release_line
auditor.report_new_for_release_line(release_line) auditor.report_new_for_release_line(release_line)
if second_pass_args.report_new_for_release_branch:
release_branch = second_pass_args.report_new_for_release_branch
auditor.report_new_for_release_branch(release_branch)
if __name__ == '__main__': if __name__ == '__main__':