Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

packaging script adjustments (#8436) 

* set encoding for license and notice scripts, split generate-license.py into generate-binary-license.py and check-licenses.py, check-licenses when -Papache-release is used

* missing docs

* doc fix

* more doc fix

* remove comments

* good catch travis +1

* fix lgtm alerts
This commit is contained in:
Clint Wylie 2019-08-29 23:27:43 -07:00 committed by GitHub
parent f55e1be80b
commit 984958122b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 277 additions and 212 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.travis.yml
View File

@ -82,7 +82,7 @@ jobs:
# (https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system). # (https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system).
- mkdir -p target - mkdir -p target
- distribution/bin/generate-license-dependency-reports.py . target --clean-maven-artifact-transfer --parallel 2 - distribution/bin/generate-license-dependency-reports.py . target --clean-maven-artifact-transfer --parallel 2
- distribution/bin/generate-license.py licenses/APACHE2 licenses.yaml LICENSES.BINARY --dependency-reports target/license-reports - distribution/bin/check-licenses.py licenses.yaml target/license-reports
- &compile_strict - &compile_strict
name: "(openjdk8) strict compilation" name: "(openjdk8) strict compilation"

7
distribution/asf-release-process-guide.md
View File

@ -152,9 +152,9 @@ If the dependency requires it, copy any licenses to the `licenses/src` or `licen
| tool | description | | tool | description |
| --- | --- | | --- | --- |
| [generate_license](bin/generate_license.py) | This script is run _automatically_ when building the distribution package to generate a `LICENSE.BINARY` file from `licenses.yaml` which is renamed to `LICENSE` in the binary package. It can also be used to verify that all dependencies are present and match the versions in `licenses.yaml` (and is run by travis for all PRs so it _should_ be up to date). | | [generate-binary-license](bin/generate-binary-license.py) | This script is run _automatically_ when building the distribution package to generate a `LICENSE.BINARY` file from `licenses.yaml` which is renamed to `LICENSE` in the binary package. |
| [generate-notice-binary](bin/generate-notice-binary.py) | This script is run _automatically_ when building the distribution package, and generates a `NOTICE.BINARY` file by appending the notice content of `licenses.yaml` to the source `NOTICE` file. This script does _not_ currently verify that all notices that need to be are present and correct, this must currently be done manually at release time if not done in the PR that changed a dependency. | | [generate-binary-notice](bin/generate-binary-notice.py) | This script is run _automatically_ when building the distribution package, and generates a `NOTICE.BINARY` file by appending the notice content of `licenses.yaml` to the source `NOTICE` file. This script does _not_ currently verify that all notices that need to be are present and correct, this must currently be done manually at release time if not done in the PR that changed a dependency. |
| [web-console/licenses](../web-console/scripts/licenses) | Updates `licenses.yaml` with all Druid of the licenses used by the Druid web-console 'binary'. | | [web-console/licenses](../web-console/script/licenses) | Updates `licenses.yaml` with all Druid of the licenses used by the Druid web-console 'binary'. |
### Additional tools ### Additional tools
@ -163,6 +163,7 @@ These additional tools were largely used to bootstrap the initial `LICENSE`, `LI
| tool | description | | tool | description |
| --- | --- | | --- | --- |
| [generate-license-dependency-reports](bin/generate-license-dependency-reports.py) | Point this to the Druid source root, and give it the location of a temp scratch directory, and it will output Maven dependency reports for Druid. (I believe I had to generate Maven dep report separately for hadoop-client) | | [generate-license-dependency-reports](bin/generate-license-dependency-reports.py) | Point this to the Druid source root, and give it the location of a temp scratch directory, and it will output Maven dependency reports for Druid. (I believe I had to generate Maven dep report separately for hadoop-client) |
| [check-licenses](bin/check-licenses.py) | Checks `licenses.yaml` against the output of `generate-license-dependency-reports.py`, used by travis and `-Papache-release` when building distribution, to verify that all dependencies are present and match the versions in `licenses.yaml`. |
| [jar-notice-lister](bin/jar-notice-lister.py) | Point this to an extracted Druid binary distribution, and give it a temp scratch directory, and it will output NOTICE information for all the Druid JAR files. | | [jar-notice-lister](bin/jar-notice-lister.py) | Point this to an extracted Druid binary distribution, and give it a temp scratch directory, and it will output NOTICE information for all the Druid JAR files. |

190
distribution/bin/generate-license.py → distribution/bin/check-licenses.py
View File

@ -22,7 +22,6 @@ import sys
from html.parser import HTMLParser from html.parser import HTMLParser
import argparse import argparse
class DependencyReportParser(HTMLParser): class DependencyReportParser(HTMLParser):
# This class parses the given html file to find all dependency reports under "Project dependencies" # This class parses the given html file to find all dependency reports under "Project dependencies"
# and "Projection transparent dependencies" sections. # and "Projection transparent dependencies" sections.
@ -208,11 +207,8 @@ class DependencyReportParser(HTMLParser):
self.license = self.compatible_license_names[data] self.license = self.compatible_license_names[data]
outfile = None def print_log_to_stderr(string):
print(string, file=sys.stderr)
def get_dep_key(group_id, artifact_id, version):
return (group_id, artifact_id, version)
def build_compatible_license_names(): def build_compatible_license_names():
compatible_licenses = {} compatible_licenses = {}
@ -272,24 +268,8 @@ def build_compatible_license_names():
compatible_licenses['-'] = '-' compatible_licenses['-'] = '-'
return compatible_licenses return compatible_licenses
def get_dep_key(group_id, artifact_id, version):
def module_to_upper(module): return (group_id, artifact_id, version)
extensions_offset = module.lower().find("extensions")
if extensions_offset < 0:
return module.upper()
elif extensions_offset == 0:
return module[0:len("extensions")].upper() + module[len("extensions"):len(module)]
else:
raise Exception("Expected extensions at 0, but {}".format(extensions_offset))
def print_outfile(string):
print(string, file=outfile)
def print_error(string):
print(string, file=sys.stderr)
def get_version_string(version): def get_version_string(version):
if type(version) == str: if type(version) == str:
@ -297,92 +277,6 @@ def get_version_string(version):
else: else:
return str(version) return str(version)
def print_license_phrase(license_phrase):
remaining = license_phrase
while len(remaining) > 0:
# print("remaining: {}".format(remaining))
# print("len: {}".format(len(remaining)))
if len(remaining) > 120:
chars_of_200 = remaining[0:120]
phrase_len = chars_of_200.rfind(" ")
if phrase_len < 0:
raise Exception("Can't find whitespace in {}".format(chars_of_200))
print_outfile(" {}".format(remaining[0:phrase_len]))
remaining = remaining[phrase_len:]
else:
print_outfile(" {}".format(remaining))
remaining = ""
def is_non_empty(dic, key):
if key in dic and dic[key] is not None:
if type(dic[key]) == str:
return len(dic[key]) > 0
else:
return True
else:
return False
def print_license(license):
license_phrase = "This product"
if license['license_category'] == "source":
license_phrase += " contains"
elif license['license_category'] == "binary":
license_phrase += " bundles"
license_phrase += " {}".format(license['name'])
if is_non_empty(license, 'version'):
license_phrase += " version {}".format(license['version'])
if is_non_empty(license, 'copyright'):
license_phrase += ", copyright {}".format(license['copyright'])
if is_non_empty(license, 'additional_copyright_statement'):
license_phrase += ", {}".format(license['additional_copyright_statement'])
if license['license_name'] != 'Apache License version 2.0':
license_phrase += " which is available under {}".format(license['license_name'])
if is_non_empty(license, 'additional_license_statement'):
license_phrase += ", {}".format(license['additional_license_statement'])
if is_non_empty(license, 'license_file_path'):
license_file_list = []
if type(license['license_file_path']) == list:
license_file_list.extend(license['license_file_path'])
else:
license_file_list.append(license['license_file_path'])
if len(license_file_list) == 1:
license_phrase += ". For details, see {}".format(license_file_list[0])
else:
license_phrase += ". For details, "
for each_file in license_file_list:
if each_file == license_file_list[-1]:
license_phrase += ", and {}".format(each_file)
elif each_file == license_file_list[0]:
license_phrase += "see {}".format(each_file)
else:
license_phrase += ", {}".format(each_file)
license_phrase += "."
print_license_phrase(license_phrase)
if 'source_paths' in license:
for source_path in license['source_paths']:
if type(source_path) is dict:
for class_name, path in source_path.items():
print_outfile(" {}:".format(class_name))
print_outfile(" * {}".format(path))
else:
print_outfile(" * {}".format(source_path))
if 'libraries' in license:
for library in license['libraries']:
if type(library) is not dict:
raise Exception("Expected dict but got {}[{}]".format(type(library), library))
if len(library) > 1:
raise Exception("Expected 1 groupId and artifactId, but got [{}]".format(library))
for group_id, artifact_id in library.items():
print_outfile(" * {}:{}".format(group_id, artifact_id))
def find_druid_module_name(dirpath): def find_druid_module_name(dirpath):
ext_start = dirpath.find("/ext/") ext_start = dirpath.find("/ext/")
if ext_start > 0: if ext_start > 0:
@ -397,7 +291,6 @@ def find_druid_module_name(dirpath):
# Druid core # Druid core
return "core" return "core"
def check_licenses(license_yaml, dependency_reports_root): def check_licenses(license_yaml, dependency_reports_root):
# Build a dictionary to facilitate comparing reported licenses and registered ones. # Build a dictionary to facilitate comparing reported licenses and registered ones.
# These dictionaries are the mapping of (group_id, artifact_id, version) to license_name. # These dictionaries are the mapping of (group_id, artifact_id, version) to license_name.
@ -411,7 +304,7 @@ def check_licenses(license_yaml, dependency_reports_root):
full_path = os.path.join(dirpath, filename) full_path = os.path.join(dirpath, filename)
# Determine if it's druid core or an extension # Determine if it's druid core or an extension
druid_module_name = find_druid_module_name(dirpath) druid_module_name = find_druid_module_name(dirpath)
print_error("Parsing {}".format(full_path)) print_log_to_stderr("Parsing {}".format(full_path))
with open(full_path) as report_file: with open(full_path) as report_file:
parser = DependencyReportParser(druid_module_name, compatible_license_names) parser = DependencyReportParser(druid_module_name, compatible_license_names)
reported_dep_to_licenses.update(parser.parse(report_file)) reported_dep_to_licenses.update(parser.parse(report_file))
@ -419,7 +312,7 @@ def check_licenses(license_yaml, dependency_reports_root):
if len(reported_dep_to_licenses) == 0: if len(reported_dep_to_licenses) == 0:
raise Exception("No dependency reports are found") raise Exception("No dependency reports are found")
print_error("Found {} reported licenses\n".format(len(reported_dep_to_licenses))) print_log_to_stderr("Found {} reported licenses\n".format(len(reported_dep_to_licenses)))
# Build registered license dictionary. # Build registered license dictionary.
registered_dep_to_licenses = {} registered_dep_to_licenses = {}
@ -468,10 +361,10 @@ def check_licenses(license_yaml, dependency_reports_root):
# If we find any mismatched license, stop immediately. # If we find any mismatched license, stop immediately.
if len(mismatched_licenses) > 0: if len(mismatched_licenses) > 0:
print_error("Error: found {} mismatches between reported licenses and registered licenses".format(len(mismatched_licenses))) print_log_to_stderr("Error: found {} mismatches between reported licenses and registered licenses".format(len(mismatched_licenses)))
for mismatched_license in mismatched_licenses: for mismatched_license in mismatched_licenses:
print_error("druid_module: {}, groupId: {}, artifactId: {}, version: {}, reported_license: {}, registered_license: {}".format(mismatched_license[0], mismatched_license[1], mismatched_license[2], mismatched_license[3], mismatched_license[4], mismatched_license[5])) print_log_to_stderr("druid_module: {}, groupId: {}, artifactId: {}, version: {}, reported_license: {}, registered_license: {}".format(mismatched_license[0], mismatched_license[1], mismatched_license[2], mismatched_license[3], mismatched_license[4], mismatched_license[5]))
print_error("") print_log_to_stderr("")
# Let's find missing licenses, which are reported but missing in the registry. # Let's find missing licenses, which are reported but missing in the registry.
for key, reported_license_druid_module in reported_dep_to_licenses.items(): for key, reported_license_druid_module in reported_dep_to_licenses.items():
@ -479,10 +372,10 @@ def check_licenses(license_yaml, dependency_reports_root):
missing_licenses.append((reported_license_druid_module[1], key[0], key[1], key[2], reported_license_druid_module[0])) missing_licenses.append((reported_license_druid_module[1], key[0], key[1], key[2], reported_license_druid_module[0]))
if len(missing_licenses) > 0: if len(missing_licenses) > 0:
print_error("Error: found {} missing licenses. These licenses are reported, but missing in the registry".format(len(missing_licenses))) print_log_to_stderr("Error: found {} missing licenses. These licenses are reported, but missing in the registry".format(len(missing_licenses)))
for missing_license in missing_licenses: for missing_license in missing_licenses:
print_error("druid_module: {}, groupId: {}, artifactId: {}, version: {}, license: {}".format(missing_license[0], missing_license[1], missing_license[2], missing_license[3], missing_license[4])) print_log_to_stderr("druid_module: {}, groupId: {}, artifactId: {}, version: {}, license: {}".format(missing_license[0], missing_license[1], missing_license[2], missing_license[3], missing_license[4]))
print_error("") print_log_to_stderr("")
# Let's find unchecked licenses, which are registered but missing in the report. # Let's find unchecked licenses, which are registered but missing in the report.
# These licenses should be checked manually. # These licenses should be checked manually.
@ -493,76 +386,27 @@ def check_licenses(license_yaml, dependency_reports_root):
unchecked_licenses.append((key[0], key[1], key[2], registered_license)) unchecked_licenses.append((key[0], key[1], key[2], registered_license))
if len(unchecked_licenses) > 0: if len(unchecked_licenses) > 0:
print_error("Warn: found {} unchecked licenses. These licenses are registered, but not found in dependency reports.".format(len(unchecked_licenses))) print_log_to_stderr("Warn: found {} unchecked licenses. These licenses are registered, but not found in dependency reports.".format(len(unchecked_licenses)))
print_error("These licenses must be checked manually.") print_log_to_stderr("These licenses must be checked manually.")
for unchecked_license in unchecked_licenses: for unchecked_license in unchecked_licenses:
print_error("groupId: {}, artifactId: {}, version: {}, reported_license: {}".format(unchecked_license[0], unchecked_license[1], unchecked_license[2], unchecked_license[3])) print_log_to_stderr("groupId: {}, artifactId: {}, version: {}, reported_license: {}".format(unchecked_license[0], unchecked_license[1], unchecked_license[2], unchecked_license[3]))
print_error("") print_log_to_stderr("")
if len(mismatched_licenses) > 0 or len(missing_licenses) > 0: if len(mismatched_licenses) > 0 or len(missing_licenses) > 0:
sys.exit(1) sys.exit(1)
def print_license_name_underbar(license_name):
underbar = ""
for _ in range(len(license_name)):
underbar += "="
print_outfile("{}\n".format(underbar))
def generate_license(apache_license_v2, license_yaml):
# Generate LICENSE.BINARY file
print_error("=== Generating the contents of LICENSE.BINARY file ===\n")
# Print Apache license first.
print_outfile(apache_license_v2)
with open(license_yaml) as registry_file:
licenses_list = list(yaml.load_all(registry_file))
# Group licenses by license_name, license_category, and then module.
licenses_map = {}
for license in licenses_list:
if license['license_name'] not in licenses_map:
licenses_map[license['license_name']] = {}
licenses_of_name = licenses_map[license['license_name']]
if license['license_category'] not in licenses_of_name:
licenses_of_name[license['license_category']] = {}
licenses_of_category = licenses_of_name[license['license_category']]
if license['module'] not in licenses_of_category:
licenses_of_category[license['module']] = []
licenses_of_module = licenses_of_category[license['module']]
licenses_of_module.append(license)
for license_name, licenses_of_name in sorted(licenses_map.items()):
print_outfile(license_name)
print_license_name_underbar(license_name)
for license_category, licenses_of_category in licenses_of_name.items():
for module, licenses in licenses_of_category.items():
print_outfile("{}/{}".format(license_category.upper(), module_to_upper(module)))
for license in licenses:
print_license(license)
print_outfile("")
print_outfile("")
if __name__ == "__main__": if __name__ == "__main__":
try: try:
parser = argparse.ArgumentParser(description='Check and generate license file.') parser = argparse.ArgumentParser(description='Check and generate license file.')
parser.add_argument('apache_license', metavar='<path to apache license file>', type=str)
parser.add_argument('license_yaml', metavar='<path to license.yaml>', type=str) parser.add_argument('license_yaml', metavar='<path to license.yaml>', type=str)
parser.add_argument('out_path', metavar='<path to output file>', type=str) parser.add_argument('dependency_reports_root', metavar='<root to maven dependency reports>', type=str)
parser.add_argument('--dependency-reports', dest='dependency_reports_root', type=str, default=None, metavar='<root to maven dependency reports>')
args = parser.parse_args() args = parser.parse_args()
with open(args.apache_license) as apache_license_file:
apache_license_v2 = apache_license_file.read()
license_yaml = args.license_yaml license_yaml = args.license_yaml
dependency_reports_root = args.dependency_reports_root dependency_reports_root = args.dependency_reports_root
with open(args.out_path, "w") as outfile:
if dependency_reports_root is not None:
check_licenses(license_yaml, dependency_reports_root) check_licenses(license_yaml, dependency_reports_root)
generate_license(apache_license_v2, license_yaml)
except KeyboardInterrupt: except KeyboardInterrupt:
print('Interrupted, closing.') print('Interrupted, closing.')

0
distribution/bin/find-missing-backports.py Normal file → Executable file
View File

184
distribution/bin/generate-binary-license.py Executable file
View File

@ -0,0 +1,184 @@
#!/usr/bin/env python3
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import yaml
import sys
import argparse
outfile = None
def print_outfile(string):
print(string, file=outfile)
def print_log_to_stderr(string):
print(string, file=sys.stderr)
def get_dep_key(group_id, artifact_id, version):
return (group_id, artifact_id, version)
def get_version_string(version):
if type(version) == str:
return version
else:
return str(version)
def module_to_upper(module):
extensions_offset = module.lower().find("extensions")
if extensions_offset < 0:
return module.upper()
elif extensions_offset == 0:
return module[0:len("extensions")].upper() + module[len("extensions"):len(module)]
else:
raise Exception("Expected extensions at 0, but {}".format(extensions_offset))
def is_non_empty(dic, key):
if key in dic and dic[key] is not None:
if type(dic[key]) == str:
return len(dic[key]) > 0
else:
return True
else:
return False
def print_license_phrase(license_phrase):
remaining = license_phrase
while len(remaining) > 0:
if len(remaining) > 120:
chars_of_200 = remaining[0:120]
phrase_len = chars_of_200.rfind(" ")
if phrase_len < 0:
raise Exception("Can't find whitespace in {}".format(chars_of_200))
print_outfile(" {}".format(remaining[0:phrase_len]))
remaining = remaining[phrase_len:]
else:
print_outfile(" {}".format(remaining))
remaining = ""
def print_license(license):
license_phrase = "This product"
if license['license_category'] == "source":
license_phrase += " contains"
elif license['license_category'] == "binary":
license_phrase += " bundles"
license_phrase += " {}".format(license['name'])
if is_non_empty(license, 'version'):
license_phrase += " version {}".format(license['version'])
if is_non_empty(license, 'copyright'):
license_phrase += ", copyright {}".format(license['copyright'])
if is_non_empty(license, 'additional_copyright_statement'):
license_phrase += ", {}".format(license['additional_copyright_statement'])
if license['license_name'] != 'Apache License version 2.0':
license_phrase += " which is available under {}".format(license['license_name'])
if is_non_empty(license, 'additional_license_statement'):
license_phrase += ", {}".format(license['additional_license_statement'])
if is_non_empty(license, 'license_file_path'):
license_file_list = []
if type(license['license_file_path']) == list:
license_file_list.extend(license['license_file_path'])
else:
license_file_list.append(license['license_file_path'])
if len(license_file_list) == 1:
license_phrase += ". For details, see {}".format(license_file_list[0])
else:
license_phrase += ". For details, "
for each_file in license_file_list:
if each_file == license_file_list[-1]:
license_phrase += ", and {}".format(each_file)
elif each_file == license_file_list[0]:
license_phrase += "see {}".format(each_file)
else:
license_phrase += ", {}".format(each_file)
license_phrase += "."
print_license_phrase(license_phrase)
if 'source_paths' in license:
for source_path in license['source_paths']:
if type(source_path) is dict:
for class_name, path in source_path.items():
print_outfile(" {}:".format(class_name))
print_outfile(" * {}".format(path))
else:
print_outfile(" * {}".format(source_path))
if 'libraries' in license:
for library in license['libraries']:
if type(library) is not dict:
raise Exception("Expected dict but got {}[{}]".format(type(library), library))
if len(library) > 1:
raise Exception("Expected 1 groupId and artifactId, but got [{}]".format(library))
for group_id, artifact_id in library.items():
print_outfile(" * {}:{}".format(group_id, artifact_id))
def print_license_name_underbar(license_name):
underbar = ""
for _ in range(len(license_name)):
underbar += "="
print_outfile("{}\n".format(underbar))
def generate_license(apache_license_v2, license_yaml):
print_log_to_stderr("=== Generating the contents of LICENSE.BINARY file ===\n")
# Print Apache license first.
print_outfile(apache_license_v2)
with open(license_yaml, encoding='utf-8') as registry_file:
licenses_list = list(yaml.load_all(registry_file))
# Group licenses by license_name, license_category, and then module.
licenses_map = {}
for license in licenses_list:
if license['license_name'] not in licenses_map:
licenses_map[license['license_name']] = {}
licenses_of_name = licenses_map[license['license_name']]
if license['license_category'] not in licenses_of_name:
licenses_of_name[license['license_category']] = {}
licenses_of_category = licenses_of_name[license['license_category']]
if license['module'] not in licenses_of_category:
licenses_of_category[license['module']] = []
licenses_of_module = licenses_of_category[license['module']]
licenses_of_module.append(license)
for license_name, licenses_of_name in sorted(licenses_map.items()):
print_outfile(license_name)
print_license_name_underbar(license_name)
for license_category, licenses_of_category in licenses_of_name.items():
for module, licenses in licenses_of_category.items():
print_outfile("{}/{}".format(license_category.upper(), module_to_upper(module)))
for license in licenses:
print_license(license)
print_outfile("")
print_outfile("")
if __name__ == "__main__":
try:
parser = argparse.ArgumentParser(description='Check and generate license file.')
parser.add_argument('apache_license', metavar='<path to apache license file>', type=str)
parser.add_argument('license_yaml', metavar='<path to license.yaml>', type=str)
parser.add_argument('out_path', metavar='<path to output file>', type=str)
args = parser.parse_args()
with open(args.apache_license, encoding="ascii") as apache_license_file:
apache_license_v2 = apache_license_file.read()
license_yaml = args.license_yaml
with open(args.out_path, "w", encoding="utf-8") as outfile:
generate_license(apache_license_v2, license_yaml)
except KeyboardInterrupt:
print('Interrupted, closing.')

7
distribution/bin/generate-notice-binary.py → distribution/bin/generate-binary-notice.py
View File

@ -51,13 +51,12 @@ def print_notice(dependency):
print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, jar, dependency['version'], dependencyHeaderLine)) print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, jar, dependency['version'], dependencyHeaderLine))
print_outfile("{}\n\n\n\n".format(notice)) print_outfile("{}\n\n\n\n".format(notice))
def generate_notice(source_notice, dependences_yaml): def generate_notice(source_notice, dependences_yaml):
print_log_to_stderr("=== Generating the contents of NOTICE.BINARY file ===\n") print_log_to_stderr("=== Generating the contents of NOTICE.BINARY file ===\n")
# Print Apache license first. # Print Apache license first.
print_outfile(source_notice) print_outfile(source_notice)
with open(dependences_yaml) as registry_file: with open(dependences_yaml, encoding='utf-8') as registry_file:
dependencies = list(yaml.load_all(registry_file)) dependencies = list(yaml.load_all(registry_file))
# Group dependencies by module # Group dependencies by module
@ -81,11 +80,11 @@ if __name__ == "__main__":
parser.add_argument('out_path', metavar='<path to output file>', type=str) parser.add_argument('out_path', metavar='<path to output file>', type=str)
args = parser.parse_args() args = parser.parse_args()
with open(args.notice) as apache_notice_file: with open(args.notice, encoding="ascii") as apache_notice_file:
source_notice = apache_notice_file.read() source_notice = apache_notice_file.read()
dependencies_yaml = args.license_yaml dependencies_yaml = args.license_yaml
with open(args.out_path, "w") as outfile: with open(args.out_path, "w", encoding="utf-8") as outfile:
generate_notice(source_notice, dependencies_yaml) generate_notice(source_notice, dependencies_yaml)
except KeyboardInterrupt: except KeyboardInterrupt:

0
distribution/bin/tag-missing-milestones.py Normal file → Executable file
View File

59
distribution/pom.xml
View File

@ -117,13 +117,13 @@
<artifactId>exec-maven-plugin</artifactId> <artifactId>exec-maven-plugin</artifactId>
<executions> <executions>
<execution> <execution>
<id>versionify-readme</id> <id>generate-readme</id>
<phase>initialize</phase> <phase>initialize</phase>
<goals> <goals>
<goal>exec</goal> <goal>exec</goal>
</goals> </goals>
<configuration> <configuration>
<executable>${project.parent.basedir}/distribution/bin/build-textfile-readme.sh</executable> <executable>${project.basedir}/bin/build-textfile-readme.sh</executable>
<arguments> <arguments>
<argument>${project.basedir}/../</argument> <argument>${project.basedir}/../</argument>
<argument>${project.parent.version}</argument> <argument>${project.parent.version}</argument>
@ -131,17 +131,17 @@
</configuration> </configuration>
</execution> </execution>
<execution> <execution>
<id>generate-license</id> <id>generate-binary-license</id>
<phase>initialize</phase> <phase>initialize</phase>
<goals> <goals>
<goal>exec</goal> <goal>exec</goal>
</goals> </goals>
<configuration> <configuration>
<executable>${project.parent.basedir}/distribution/bin/generate-license.py</executable> <executable>${project.basedir}/bin/generate-binary-license.py</executable>
<arguments> <arguments>
<argument>${project.basedir}/../licenses/APACHE2</argument> <argument>${project.parent.basedir}/licenses/APACHE2</argument>
<argument>${project.basedir}/../licenses.yaml</argument> <argument>${project.parent.basedir}/licenses.yaml</argument>
<argument>${project.basedir}/../LICENSE.BINARY</argument> <argument>${project.parent.basedir}/LICENSE.BINARY</argument>
</arguments> </arguments>
</configuration> </configuration>
</execution> </execution>
@ -152,11 +152,11 @@
<goal>exec</goal> <goal>exec</goal>
</goals> </goals>
<configuration> <configuration>
<executable>${project.parent.basedir}/distribution/bin/generate-notice-binary.py</executable> <executable>${project.basedir}/bin/generate-binary-notice.py</executable>
<arguments> <arguments>
<argument>${project.basedir}/../NOTICE</argument> <argument>${project.parent.basedir}/NOTICE</argument>
<argument>${project.basedir}/../licenses.yaml</argument> <argument>${project.parent.basedir}/licenses.yaml</argument>
<argument>${project.basedir}/../NOTICE.BINARY</argument> <argument>${project.parent.basedir}/NOTICE.BINARY</argument>
</arguments> </arguments>
</configuration> </configuration>
</execution> </execution>
@ -276,6 +276,43 @@
<id>apache-release</id> <id>apache-release</id>
<build> <build>
<plugins> <plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>generate-licenses-report</id>
<phase>initialize</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${project.basedir}/bin/generate-license-dependency-reports.py</executable>
<arguments>
<argument>${project.basedir}/../</argument>
<argument>${project.basedir}/target</argument>
<argument>--clean-maven-artifact-transfer</argument>
<argument>--parallel</argument>
<argument>2</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>check-licenses</id>
<phase>test</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${project.basedir}/bin/check-licenses.py</executable>
<arguments>
<argument>${project.parent.basedir}/licenses.yaml</argument>
<argument>${project.basedir}/target/license-reports</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId> <artifactId>maven-assembly-plugin</artifactId>