lucene/dev-tools/scripts/svnBranchToGit.py

798 lines
30 KiB
Python
Raw Normal View History

from __future__ import print_function
"""
To be done:
- Investigate whether it is possible to obtain the last svn revision number without switching to it.
- Investigate file mode differences reported by gitk, see svn revision 171449.
- simplify difference check to a single call to diff.
Verify that all common files are equal, ignore non common files, check stderr and stdout of diff.
"""
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Workaround for slow updates from an svn branch to git.
See also jira issue INFRA-9182
Situation:
Remote svn repo ---> (slow) git-svn fetch ---> Remote git repo (upstream)
| |
| |
v v
Local svn working copy ---> this workaround ---> Local git repo
When the remote git-svn fetch is slow, the remote git repo is behind
the remote svn repo.
When this script is run it will first check that the local working copy and repository are clean.
Then it switches the svn working copy to the branch, which updates from the remote.
Then it fetches the branch from the git upstream repo, and merges the branch locally.
Normally the local svn and git will then be at the same svn revision, and the script will exit.
Otherwise the remote git repo is out of date, and the following happens.
It is checked that the hostname and path and the uuid of the remote svn repo
as reported by the local svn working copy and as reported by the local git repo
are the same.
For the branch branchname in a local git repository following an upstream git-svn git repository,
this maintains commits on a temporary git branch branchname.svn in the local git repository.
These commits contain metdata that differs slightly from git svn (svn2git-id: instead of git-svn-id:).
Otherwise the messages of the added commits are the same as their counterparts from git svn,
except occasionally for an added or missed empty line when the svn commit message ends in new line.
Normally the added git commits and their git-svn counterparts have no differences between their working trees.
However such differences can occur, for example occasionally file modes are different in the git working tree.
See also the documentation of git-svn reset and the limitations below.
In order not to interfere with git-svn this script only adds commits to a temporary branch
branchname.svn, and the commit messages are chosen differently, they do not contain git-svn-id: .
In case an earlier branchname.svn exists, it will first be deleted if necessary,
and restarted at the later branch.
Therefore branchname.svn is temporary and should only be used locally.
By default, no more than 20 commits will be added to branchname.svn in a single run.
The earlier revision number is taken from the git-svn-id: message of git svn,
or from the latest revision number in the commit message on branchname.svn,
whichever is later.
This allows branchname.svn to be used as a local git branch instead of branchname
to develop new features locally, for example by merging branchname.svn into a feature branch.
This works by interpretation of the lines of svn update messages (U/A/D etc.)
by copying these files and their protection bits from the local svn working copy into the git working tree,
and by deleting files and directories in the git working tree.
An example commit in lucene-solr that adds a binary file, on which this script provides a correct git working tree:
svn revision 1707457
git-svn commit 3c0390f71e1f08a17f32bc207b4003362f8b6ac2
Limitations:
All svn properties are ignored here.
Commit messages added to the git repo occasionally do not have the same number of empty lines
as the corresponding svn commit message.
"""
""" This was developed on Linux using the following program versions:
python 2.7.6
python 3.4.3
git 1.9.1
svn 1.8.8
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
sed (GNU sed) 4.2.2
grep (GNU grep) 2.16
diff (GNU diffutils) 3.3
cp (GNU coreutils) 8.21
rm (GNU coreutils) 8.21
mkdir (GNU coreutils) 8.21
gitk (part of git) was used for manual testing:
- delete branchname.svn, reset branchname.svn and branchname to earlier to simulate going back in history,
- diff a commit generated here to a commit from git svn, ideally there are no differences,
- update, reload, show commits in reverse order of commit date, ...
"""
import os
import subprocess
import shutil
from xml import sax
from xml.sax.handler import ContentHandler
try:
from urllib.parse import urlparse # python 3
except ImportError:
from urlparse import urlparse # python 2
import sys
binaryToString = sys.version_info >= (3, 0)
def decodeBytesToString(bytes):
return bytes.decode("utf-8")
class SvnInfoHandler(ContentHandler):
commitTag = "commit"
revisionAttr = "revision"
urlTag = "url"
uuidTag = "uuid"
charCollectTags = (urlTag, uuidTag) # also used as SvnInfoHandler attributes
def __init__(self):
self.lastChangeRev = None
self.lastLogEntry = None
for tag in self.charCollectTags:
setattr(self, tag, None)
self.chars = None
def startElement(self, name, attrs):
if name == self.commitTag:
self.lastChangeRev = int(attrs.getValue(self.revisionAttr))
elif name in self.charCollectTags:
self.chars = ""
def characters(self, content):
if self.chars is not None:
self.chars += content
def endElement(self, name):
if name in self.charCollectTags:
chars = self.chars
setattr(self, name, chars)
self.chars = None
def getLastChangeRevision(self):
return self.lastChangeRev
class SvnLogEntry(object):
pass # attributes set in SvnLogHandler: revision, author, date, msg
class SvnLogHandler(ContentHandler): # collect list of SvnLogEntry's
logEntryTag = "logentry"
revisionAttr = "revision" # also used as SvnLogEntry attribute
authorTag = "author"
dateTag = "date"
msgTag = "msg"
charCollectTags = (authorTag, dateTag, msgTag) # also used as SvnLogEntry attributes
def __init__(self):
self.logEntries = []
self.chars = None
def startElement(self, name, attrs):
if name == self.logEntryTag:
self.lastLogEntry = SvnLogEntry()
setattr(self.lastLogEntry, self.revisionAttr, int(attrs.getValue(self.revisionAttr)))
for tag in self.charCollectTags:
setattr(self.lastLogEntry, tag, None)
return
if name in self.charCollectTags:
self.chars = ""
def characters(self, content):
if self.chars is not None:
self.chars += content
def endElement(self, name):
if name in self.charCollectTags:
chars = self.chars
setattr(self.lastLogEntry, name, chars)
self.chars = None
return
if name == self.logEntryTag:
self.logEntries.append(self.lastLogEntry)
self.lastLogEntry = None
def getLogEntries(self):
return self.logEntries
class SubProcessAtPath(object):
def __init__(self, pathName, verbose=True):
self.pathName = pathName
self.verbose = verbose
def getPathName(self):
return self.pathName
def chDirToPath(self):
if self.pathName != os.getcwd():
os.chdir(self.pathName)
assert self.pathName == os.getcwd()
def __str__(self):
return self.__class__.__name__ + "(" + self.pathName + ")"
def checkCall(self, *args, **kwArgs):
self.chDirToPath()
if self.verbose:
print("check_call args:", " ".join(*args), str(**kwArgs))
subprocess.check_call(*args, **kwArgs)
def checkOutput(self, *args, **kwArgs):
self.chDirToPath()
if self.verbose:
print("check_output args:", " ".join(*args), str(**kwArgs))
result = subprocess.check_output(*args, **kwArgs)
if self.verbose:
print("check_output result:", result)
return result
def checkOutputAsStr(self, *args, **kwArgs):
self.chDirToPath()
if self.verbose:
print("check_output args:", " ".join(*args), str(**kwArgs))
result = subprocess.check_output(*args, **kwArgs)
if binaryToString:
result = decodeBytesToString(result)
if self.verbose:
print("check_output result:", result)
return result
def nonEmptyLines(text):
return [line for line in text.split("\n") if len(line) > 0]
class SvnWorkingCopy(SubProcessAtPath):
def __init__(self, pathName):
SubProcessAtPath.__init__(self, pathName, verbose=False)
self.url = None
self.uuid = None
svnCmd = "svn"
def ensureNoLocalModifications(self):
localMods = self.checkOutputAsStr((self.svnCmd, "status"))
if localMods:
errorExit(self, "should not have local modifications:\n", localMods)
def updateOutput(self, revision):
result = self.checkOutputAsStr((self.svnCmd, "update", "-r", str(revision)))
return result
def switch(self, repoBranchName):
self.checkCall((self.svnCmd, "switch", ("^/" + repoBranchName), "--ignore-ancestry"))
def parseInfo(self):
infoXml = self.checkOutput((self.svnCmd, "info", "--xml")) # bytes in python 3.
infoHandler = SvnInfoHandler()
sax.parseString(infoXml, infoHandler)
self.uuid = infoHandler.uuid
self.url = infoHandler.url
self.lastChangeRev = infoHandler.getLastChangeRevision()
def getUrl(self):
if self.url == None:
self.parseInfo()
return self.url
def getUuid(self):
if self.uuid == None:
self.parseInfo()
return self.uuid
def lastChangedRevision(self):
self.parseInfo()
return self.lastChangeRev
def getLogEntries(self, fromRevision, toRevision, maxNumLogEntries):
revRange = self.revisionsRange(fromRevision, toRevision)
logXml = self.checkOutput((self.svnCmd, "log", "-r", revRange, "--xml", "-l", str(maxNumLogEntries)))
logHandler = SvnLogHandler()
sax.parseString(logXml, logHandler)
return logHandler.getLogEntries()
def revisionsRange(self, fromRevision, toRevision):
return str(fromRevision) + ":" + str(toRevision)
class GitRepository(SubProcessAtPath):
def __init__(self, pathName):
SubProcessAtPath.__init__(self, pathName, verbose=False)
self.currentBranch = None
gitCmd = "git"
def checkOutBranch(self, branchName):
self.checkCall((self.gitCmd, "checkout", branchName))
self.currentBranch = branchName
def getCurrentBranch(self):
if self.currentBranch is None:
gitStatusOut = self.checkOutputAsStr((self.gitCmd, "status"))
if gitStatusOut.startswith("On branch "):
self.currentBranch = gitStatusOut.split()[2] # works also without () ???
else:
errorExit(self, "not on a branch:", gitStatusOut)
return self.currentBranch
def workingDirectoryClean(self):
gitStatusOut = self.checkOutputAsStr((self.gitCmd, "status"))
expSubString = "nothing to commit, working directory clean"
return gitStatusOut.find(expSubString) >= 0
def listBranches(self, pattern):
result = self.checkOutputAsStr((self.gitCmd, "branch", "--list", pattern))
return result
def branchExists(self, branchName):
listOut = self.listBranches(branchName) # CHECKME: using branchName as pattern may not always be ok.
return len(listOut) > 0
def deleteBranch(self, branchName):
self.checkCall((self.gitCmd, "branch", "-D", branchName))
if branchName == self.currentBranch:
self.currentBranch = None
def createBranch(self, branchName):
self.checkCall((self.gitCmd, "branch", branchName))
def fetch(self, upStream):
self.checkCall((self.gitCmd, "fetch", upStream))
def merge(self, branch, fromBranch):
self.checkCall((self.gitCmd, "merge", branch, fromBranch))
def getCommitMessage(self, commitRef):
result = self.checkOutputAsStr((self.gitCmd, "log", "--format=%B", "-n", "1", commitRef))
return result
def getCommitAuthorName(self, commitRef):
result = self.checkOutputAsStr((self.gitCmd, "log", "--format=%aN", "-n", "1", commitRef))
return result
def getCommitAuthorEmail(self, commitRef):
result = self.checkOutputAsStr((self.gitCmd, "log", "--format=%aE", "-n", "1", commitRef))
return result
def getLatestCommitForAuthor(self, svnAuthor):
# print('Get git commit for author "%s, type=%s"' % (svnAuthor, str(type(svnAuthor))))
authorCommit = self.checkOutputAsStr(
" ".join((self.gitCmd, "rev-list", "--all", "-i", ("--author=" + svnAuthor), # see git commit documentation on --author
"|", # pipe should have a buffer for at most a few commit ids.
"head", "-1" # the first line
)),
shell=True) # use shell pipe
authorCommit = authorCommit.rstrip("\n")
return authorCommit
gitSvnMarker = "git-svn-id:" # added and used by git svn dcommit
svn2gitMarker = "svn2git-id:" # added and used here.
def getSvnRemoteUuidRevisionFromCommitMessage(self, commitMessage, marker):
words = commitMessage.split()
if not marker in words:
return (None, None, None)
svnId = words[words.index(marker) + 1]
splitSvnId = svnId.split("@")
svnRemote = splitSvnId[0]
svnRevision = int(splitSvnId[1])
svnRepoUuid = words[words.index(marker) + 2]
return (svnRemote, svnRepoUuid, svnRevision)
def getSvnRemoteAndUuidAndRevision(self, gitSvnCommitRef):
gitSvnCommitMessage = self.getCommitMessage(gitSvnCommitRef)
return self.getSvnRemoteUuidRevisionFromCommitMessage(gitSvnCommitMessage, self.gitSvnMarker)
def lastTempGitSvnRevision(self, tempBranchCommitRef): # at a commit generated here on the temp branch.
gitCommitMessage = self.getCommitMessage(tempBranchCommitRef)
(svnRemote, svnRepoUuid, svnRevision) = self.getSvnRemoteUuidRevisionFromCommitMessage(gitCommitMessage, self.svn2gitMarker)
return svnRevision
def addAllToIndex(self):
self.checkCall((self.gitCmd, "add", "-A", self.getPathName()))
def commit(self, message,
authorName, authorEmail, authorDate,
committerName, committerEmail, committerDate):
author = ''.join((authorName, " <", authorEmail, ">"))
os.environ["GIT_COMMITTER_NAME"] = committerName # no need to save/restore earlier environment state.
os.environ["GIT_COMMITTER_EMAIL"] = committerEmail
os.environ["GIT_COMMITTER_DATE"] = committerDate
self.checkCall((self.gitCmd, "commit",
"--allow-empty", # in case only svn poperties changed.
("--message=" + message),
("--author=" + author),
("--date=" + authorDate) ))
def cleanDirsForced(self):
self.checkCall((self.gitCmd, "clean", "-fd")) # Use -fdx to also remove ignored files.
def errorExit(*messageParts):
raise RuntimeError(" ".join(map(str, messageParts)))
def allSuccessivePairs(lst):
return [lst[i:i+2] for i in range(len(lst)-1)]
def octal(mode):
return format(mode, 'o')
def checkEqualProtectionBits(fn1, fn2):
stat1 = os.stat(fn1)
stat2 = os.stat(fn2)
if stat1.st_mode != stat2.st_mode:
print("Protection bits %s of %s" % (octal(stat1.st_mode), fn1))
print("Protection bits %s of %s" % (octal(stat2.st_mode), fn2))
return False
return True
def verifyGitFilesAgainstSvn(gitRepo, svnWorkingCopy):
# The files under version control at the git repo can be enumerated quickly by: git ls-tree -r trunk.svn | cut --fields=2-
# This makes sense because all files, including binary files, are added.
# svn ls -R is too slow to use here (this lists about 12 file names per second, lucene-solr has well over 4000).
fileNamesOut = gitRepo.checkOutputAsStr((gitRepo.gitCmd, "ls-tree", "-r", "--name-only", gitRepo.getCurrentBranch()))
fileNames = nonEmptyLines(fileNamesOut)
print("verifyGitFilesAgainstSvn checking", len(fileNames), "files")
result = True
for fileName in fileNames:
#print("fileName", fileName)
fileNameInGitRepo = os.path.join(gitRepo.getPathName(), fileName)
#print("fileNameInGitRepo", fileNameInGitRepo)
fileNameInSvnWorkingCopy = os.path.join(svnWorkingCopy.getPathName(), fileName)
#print("fileNameInSvnWorkingCopy", fileNameInSvnWorkingCopy)
try:
diffOutput = subprocess.check_output(("diff", "-q", fileNameInGitRepo, fileNameInSvnWorkingCopy))
except (subprocess.CalledProcessError, exitError):
print("difference in file", fileName)
print("diff exitError", exitError)
result = False
if not checkEqualProtectionBits(fileNameInSvnWorkingCopy, fileNameInGitRepo):
result = False
if result:
print("no differences")
else:
print("some differences")
"""
On clean checkouts of both svn and git the command:
diff -r svndir gitdir
reports only .svn .git and empty directories in the svn working copy, for example:
Only in ./svnwork/lucene-solr/lucene/analysis/icu: lib
This diff output could be checked here.
To clean an svn working copy:
rm -r * # also .hgignore .caches, all except .svn
svn update # this is a local svn operation
To clean a git working directory:
rm -r * # all except .git
git checkout branchname -- .
"""
def deleteEmptyDirs(pathName, topDirName):
""" Delete higher level directories of pathName when empty, but do not delete topDirName """
head, tail = os.path.split(pathName)
while (head != topDirName) and not os.listdir(head):
assert head.startswith(topDirName) # , topDirName + " <<>> " + head
# subprocess.check_call(("rm", "-r", head)) # delete empty directory
os.rmdir(head)
head, tail = os.path.split(head)
def setGitWorkingTreeViaSvnCheckout(svnWorkingCopy, revision, gitRepo):
svnUpdateOutputLines = svnWorkingCopy.updateOutput(revision)
""" Some example lines:
U solr/solrj/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
U solr/core
Updated to revision 1707390.
From svn help update:
For each updated item a line will be printed with characters reporting
the action taken. These characters have the following meaning:
A Added
D Deleted
U Updated
C Conflict
G Merged
E Existed
R Replaced
Characters in the first column report about the item itself.
Characters in the second column report about properties of the item.
A 'B' in the third column signifies that the lock for the file has
been broken or stolen.
A 'C' in the fourth column indicates a tree conflict, while a 'C' in
the first and second columns indicate textual conflicts in files
and in property values, respectively.
"""
for svnUpdateLine in nonEmptyLines(svnUpdateOutputLines):
if svnUpdateLine.startswith("Updating "): # first line
continue
if svnUpdateLine.startswith("Updated to"): # last line
revisionStr = svnUpdateLine.split()[3][:-1]
assert revision == int(revisionStr), revisionStr
continue
print(svnUpdateLine)
itemChar = svnUpdateLine[0]
itemPropChar = svnUpdateLine[1]
lockChar = svnUpdateLine[2]
treeConflictChar = svnUpdateLine[3]
fileName = svnUpdateLine[5:]
validItemChars = (" ", "A", "D", "U")
assert itemChar in validItemChars, "revision %d itemChar %s, fileName %s" % (revision, itemChar, fileName)
assert itemPropChar in validItemChars, "revision %d itemPropChar %s, working copy not clean fileName %s" % (revision, itemPropChar, fileName)
assert lockChar == " ", "revision %d lockChar %s fileName %s" % (revision, lockChar, fileName)
assert treeConflictChar == " ", "revision %d treeConflictChar %s fileName %s" % (revision, treeConflictChar, fileName)
fileNameInSvnWorkingCopy = os.path.join(svnWorkingCopy.getPathName(), fileName)
fileNameInGitRepo = os.path.join(gitRepo.getPathName(), fileName)
setFileProtectionBits = False
if itemChar == "D": # deleted in svn working copy
if os.path.isdir(fileNameInGitRepo):
print("Deleting directory %s" % fileNameInGitRepo)
# subprocess.check_call(("rm", "-r", fileNameInGitRepo)) # delete in git working tree
shutil.rmtree(fileNameInGitRepo) # delete completely in git working tree
deleteEmptyDirs(fileNameInGitRepo, gitRepo.getPathName()) # delete empty dirs in git repo
elif os.path.isfile(fileNameInGitRepo):
print("Deleting file %s" % fileNameInGitRepo)
# subprocess.check_call(("rm", fileNameInGitRepo)) # delete in git working tree
os.remove(fileNameInGitRepo)
deleteEmptyDirs(fileNameInGitRepo, gitRepo.getPathName())
else:
print("Non deleting non existing file %s" % fileName)
elif itemChar in ("A", "U"): # added or updated in svn working copy
if os.path.isdir(fileNameInSvnWorkingCopy):
if not os.path.isdir(fileNameInGitRepo):
print("Creating directory %s" % fileName)
#subprocess.check_call(("mkdir", fileNameInGitRepo)) # new directory in git working tree
os.mkdir(fileNameInGitRepo)
else:
print("Not creating existing directory %s" % fileName)
elif os.path.isfile(fileNameInSvnWorkingCopy):
head, tail = os.path.split(fileNameInGitRepo)
if not os.path.isdir(head):
print("Creating directory for file %s" % fileNameInGitRepo)
os.mkdir(head)
# print("Copying file %s" % fileName # Common case)
# subprocess.check_call(("cp", fileNameInSvnWorkingCopy, fileNameInGitRepo)) # copy into git working tree
shutil.copyfile(fileNameInSvnWorkingCopy, fileNameInGitRepo)
setFileProtectionBits = True
else:
assert False, "Cannot add or update non existing file %s" % fileNameInSvnWorkingCopy
else:
assert itemChar == " " # nothing to do
if itemPropChar != " ":
print("At revision %d ignoring svn property change type %s for file %s" % (revision, itemPropChar, fileName))
setFileProtectionBits = True # svn:executable may have been set or unset.
if setFileProtectionBits:
if os.path.isfile(fileNameInSvnWorkingCopy):
statSvn = os.stat(fileNameInSvnWorkingCopy)
statGit = os.stat(fileNameInGitRepo)
if statSvn.st_mode != statGit.st_mode:
print("Changing mode from %s to %s for %s" % (octal(statGit.st_mode), octal(statSvn.st_mode), fileNameInGitRepo))
os.chmod(fileNameInGitRepo, statSvn.st_mode)
def assertUrlsSameExceptScheme(url1, url2): # may only differ by scheme http:// or https://
scheme1, netloc1, path1, params1, query1, fragment1 = urlparse(url1)
scheme2, netloc2, path2, params2, query2, fragment2 = urlparse(url2)
#print(scheme1, netloc1, path1, params1, query1, fragment1)
#print(scheme2, netloc2, path2, params2, query2, fragment2)
assert netloc1 == netloc2
assert path1 == path2
assert params1 == params2
assert query1 == query2
assert fragment1 == fragment2
def maintainTempGitSvnBranch(branchName, tempGitBranchName,
svnWorkingCopyOfBranchPath, svnRepoBranchName,
gitRepoPath, gitUpstream,
maxCommits=20, # generate at most this number of commits on tempGitBranchName, rerun to add more.
testMode=False):
assert maxCommits >= 1
gitRepo = GitRepository(gitRepoPath)
gitRepo.checkOutBranch(branchName) # fails with git message when working directory is not clean
svnWorkingCopy = SvnWorkingCopy(svnWorkingCopyOfBranchPath)
svnWorkingCopy.ensureNoLocalModifications()
svnWorkingCopy.switch(svnRepoBranchName) # switch to repo branch, update to latest revision
lastSvnRevision = svnWorkingCopy.lastChangedRevision() # int to allow comparison
#print(svnWorkingCopy, "lastSvnRevision:", lastSvnRevision)
gitRepo.fetch(gitUpstream)
if testMode:
pass # leave branch where it is, as if the last commits from upstream did not arrive
else:
gitRepo.merge(branchName, gitUpstream + "/" + branchName)
(gitSvnRemote, gitSvnRepoUuid, lastSvnRevisionOnGitSvnBranch) = gitRepo.getSvnRemoteAndUuidAndRevision(branchName)
svnUrl = svnWorkingCopy.getUrl()
svnRepoUuid = svnWorkingCopy.getUuid()
print("gitSvnRemote:", gitSvnRemote)
print("svnUrl:", svnUrl)
print("svn repo uuid:", svnRepoUuid)
assertUrlsSameExceptScheme(gitSvnRemote, svnUrl)
assert gitSvnRepoUuid == svnRepoUuid
# check whether tempGitBranchName exists:
diffBaseRevision = lastSvnRevisionOnGitSvnBranch
svnTempRevision = None
doCommitOnExistingTempBranch = False
if gitRepo.branchExists(tempGitBranchName):
print(tempGitBranchName, "exists")
# update lastSvnRevisionOnGitSvnBranch from there.
svnTempRevision = gitRepo.lastTempGitSvnRevision(tempGitBranchName)
if svnTempRevision is None:
print("Warning: no svn revision found on branch:", tempGitBranchName)
else:
if svnTempRevision > lastSvnRevisionOnGitSvnBranch:
diffBaseRevision = svnTempRevision
doCommitOnExistingTempBranch = True
gitRepo.checkOutBranch(tempGitBranchName)
if lastSvnRevision == diffBaseRevision:
print(gitRepo, gitRepo.getCurrentBranch(), "up to date with", svnWorkingCopy, svnRepoBranchName)
verifyGitFilesAgainstSvn(gitRepo, svnWorkingCopy)
return
if lastSvnRevision < diffBaseRevision: # unlikely, do nothing
print(gitRepo, gitRepo.getCurrentBranch(), "later than", svnWorkingCopy, ", nothing to update.")
return
print(gitRepo, gitRepo.getCurrentBranch(), "earlier than", svnWorkingCopy)
if not gitRepo.workingDirectoryClean():
errorExit(gitRepo, "on branch", gitRepo.getCurrentBranch(), "not clean")
print(gitRepo,"on branch", gitRepo.getCurrentBranch(), "and clean")
if not doCommitOnExistingTempBranch: # restart temp branch from branch
assert gitRepo.getCurrentBranch() == branchName
if gitRepo.branchExists(tempGitBranchName): # tempGitBranchName exists, delete it first.
print("Branch", tempGitBranchName, "exists, deleting")
gitRepo.deleteBranch(tempGitBranchName)
if gitRepo.branchExists(tempGitBranchName):
errorExit("Could not delete branch", tempGitBranchName, "from", gitRepo)
gitRepo.createBranch(tempGitBranchName)
gitRepo.checkOutBranch(tempGitBranchName)
print("Started branch", tempGitBranchName, "at", branchName)
assert gitRepo.getCurrentBranch() == tempGitBranchName
maxNumLogEntries = maxCommits + 1
svnLogEntries = svnWorkingCopy.getLogEntries(diffBaseRevision, lastSvnRevision, maxNumLogEntries)
numCommits = 0
startRevision = svnLogEntries[0].revision
ignore = svnWorkingCopy.updateOutput(startRevision)
for (logEntryFrom, logEntryTo) in allSuccessivePairs(svnLogEntries):
setGitWorkingTreeViaSvnCheckout(svnWorkingCopy, logEntryTo.revision, gitRepo)
gitRepo.addAllToIndex() # add all changes from the git working tree to the git index.
# commit, put toRevision at end so it can be picked up later.
commitMessageMetaData = gitRepo.svn2gitMarker + " " + gitSvnRemote + "@" + str(logEntryTo.revision) + " " + gitSvnRepoUuid
# git-svn adds this commit metadata:
# git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1719562 13f79535-47bb-0310-9956-ffa450edef68
# This script uses svn2git-id: instead of git-svn-id:
message = logEntryTo.msg + "\n\n" + commitMessageMetaData
authorCommit = gitRepo.getLatestCommitForAuthor(logEntryTo.author)
authorName = gitRepo.getCommitAuthorName(authorCommit)
authorEmail = gitRepo.getCommitAuthorEmail(authorCommit)
# print("Author name and email:", authorName, authorEmail)
gitRepo.commit(message,
authorName, authorEmail, logEntryTo.date,
authorName, authorEmail, logEntryTo.date) # author is also git committer, just like git-svn
numCommits += 1
#print("Commit author:", logEntryTo.author)
print("Commit date:", logEntryTo.date)
#print("Commit message:", logEntryTo.msg)
gitRepo.cleanDirsForced() # delete untracked directories and files
if not gitRepo.workingDirectoryClean():
errorExit(gitRepo, "on branch", gitRepo.getCurrentBranch(), "not clean, numCommits:", numCommits)
diffBaseRevision = logEntryTo.revision
print('') # show empty line after commit info
print("Added", numCommits, "commit(s) to branch", tempGitBranchName)
if lastSvnRevision == diffBaseRevision:
print(gitRepo, gitRepo.getCurrentBranch(), "up to date with", svnWorkingCopy, svnRepoBranchName)
verifyGitFilesAgainstSvn(gitRepo, svnWorkingCopy)
return
if __name__ == "__main__":
testMode = False # when true, leave branch where it is, as if the last commits from upstream did not arrive
defaultMaxCommits = 20
maxCommits = defaultMaxCommits
import sys
argv = sys.argv[1:]
while argv:
if argv[0] == "test":
testMode = True
else:
try:
maxCommits = int(argv[0])
assert maxCommits >= 1
except:
errorExit("Argument(s) [test] [maximum number of commits], defaults are false and " + defaultMaxCommits)
argv = argv[1:]
repo = "lucene-solr"
branchName = "trunk"
tempGitBranchName = branchName + ".svn"
home = os.path.expanduser("~")
svnWorkingCopiesPath = os.path.join(home, "svnwork")
gitReposPath = os.path.join(home, "gitrepos")
if sys.argv[0].startswith(svnWorkingCopiesPath) or sys.argv[0].startswith(gitReposPath):
errorExit(sys.argv[0] + " cannot run from svn working copy or git working tree, copy to another place and run from there.")
svnWorkingCopyOfBranchPath = os.path.join(svnWorkingCopiesPath, repo)
svnRepoBranchName = "lucene/dev/" + branchName # for svn switch to
gitRepoPath = os.path.join(gitReposPath, repo)
gitUpstream = "upstream"
maintainTempGitSvnBranch(branchName, tempGitBranchName,
svnWorkingCopyOfBranchPath, svnRepoBranchName,
gitRepoPath, gitUpstream,
maxCommits=maxCommits,
testMode=testMode)