Describe the mirroring protocol. Drop URLs that don't

need to be mirrored.
This commit is contained in:
Martin v. Löwis 2010-07-04 21:33:54 +00:00
parent dbba728614
commit 73f32496bd
1 changed files with 22 additions and 14 deletions

View File

@ -87,14 +87,12 @@ Examples:
Special pages a mirror needs to provide
=======================================
A mirror is a strict copy of PyPI, so it provides the same structure
A mirror is a subset copy of PyPI, so it provides the same structure
by copying it.
- pypi: html version of the package index
- simple: rest version of the package index
- packages: packages, stored by Python version, and letters
- stats : statistics on downloads
- XXX
- serversig: signatures for the simple pages
It also needs to provide two specific elements:
@ -166,9 +164,19 @@ relevant links, plus a small part about `User-Agent`.
The mirroring protocol
::::::::::::::::::::::
XXX Need to describe the protocol here.
Mirrors must reduce the amount of data transfered between the central
server and the mirror. To achieve that, they MUST use the changelog()
PyPI XML-RPC call, and only refetch the packages that have been
changed since the last time. For each package P, they MUST copy
documents /simple/P/ and /serversig/P. If a package is deleted on the
central server, they MUST delete the package and all associated files.
To detect modification of package files, they MAY cache the file's
ETag, and MAY request skipping it using the If-none-match header.
The z3c.pypimirror package [#zcpkg]_ provides an application that
Each mirroring tool MUST identify itself using a descripte User-agent
header.
The pep381client package [#pep381client]_ provides an application that
respects this protocol to browse PyPI.
User-agent request header
@ -190,13 +198,13 @@ How a client can use PyPI and its mirrors
:::::::::::::::::::::::::::::::::::::::::
Clients that are browsing PyPI should be able to use alternative
mirrors, by getting the list of the mirrors using `mirrors.pypi.python.org`.
mirrors, by getting the list of the mirrors using `last.pypi.python.org`.
Code example::
>>> import socket
>>> socket.gethostbyname_ex('mirrors.pypi.python.org')[-1]
['82.94.164.163', '88.191.64.248']
>>> socket.gethostbyname_ex('last.pypi.python.org')[0]
'h.pypi.python.org'
The clients so far that could use this mechanism:
@ -268,8 +276,8 @@ References
.. [#pep305]
http://www.python.org/dev/peps/pep-0305/#id19
.. [#zcpkg]
http://pypi.python.org/pypi/z3c.pypimirror
.. [#pep381client]
http://pypi.python.org/pypi/pep381client
.. [#iso8601]
http://en.wikipedia.org/wiki/ISO_8601
@ -290,10 +298,10 @@ References
http://www.chrisarndt.de/projects/eggbasket
Aknowledgments
==============
Acknowledgments
===============
Martin von Loewis, Georg Brandl.
Georg Brandl.
Copyright