iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

various fixes, more details

This commit is contained in:
Tarek Ziadé 2009-03-30 04:53:39 +00:00
parent f2cb34d7a7
commit e555435799
1 changed files with 83 additions and 64 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

147
pep-0381.txt
View File

@ -10,6 +10,7 @@ Created: 21-March-2009
Python-Version: N.A.
Post-History:
Abstract
========
@ -35,7 +36,6 @@ In order to make the system more reliable, this PEP describes:
by PyPI, in order to get hit counts and the last modified date.
- how a mirror should synchronize with PyPI
- how a client can implement a fail-over mechanism
- a contact form for Package maintainers
Mirror listing and registering
@ -46,31 +46,59 @@ When a mirror is proposed on the mailing list, it is manually
added in a mirror list in the PyPI application after it
has been checked to be compliant with the mirroring rules.
A mirror has to be a hostname. For example:
- http://pypi.my-company.com
- http://mirror-pypi.somewhere.org
The mirror list is handled by a DNS entry for this hostname:
mirrors.pypi.python.org
When a mirror is added into the DNS, it becomes an official
IP for `mirrors.pypi.python.org`.
IP for `mirrors.pypi.python.org`, and requests will be sent
to the given IP. Therefore the mirror maintainer should not
change the IP provided. If the IP has to change for any reason,
the mirror maintainer has to send a mail to catalog-SIG at least
one week before the change so the DNS entry can be changed on time.
The new mirror also appears at `http://pypi.python.org/mirrors`
which is a human-readable page that gives the list of mirrors.
This page also explains how to register a new mirror.
Statistics page
:::::::::::::::
PyPI provides statistics on downloads at `/stats`. This page is
calculated daily by PyPI, by reading all mirrors' local stats and
summing them.
The stats are presented in daily or montly files, under `/stats/days`
and `/stats/months`. Each file is a `bzip2` file with these formats:
- YYYY-MM-DD.bz2 for daily files
- YYYY-MM.bz2 for monthly files
Examples:
- /stats/days/2008-11-06.bz2
- /stats/days/2008-11-07.bz2
- /stats/days/2008-11-08.bz2
- /stats/months/2008-11.bz2
- /stats/months/2008-10.bz2
Special pages a mirror needs to provide
=======================================
A mirror needs to provide four pages, beside the index one:
A mirror is a strict copy of PyPI, so it provides the same structure
by copying it.
- pypi: html version of the package index
- simple: rest version of the package index
- packages: packages, stored by Python version, and letters
- stats : statistics on downloads
- XXX
It also needs to provide two specific elements:
- last-modified
- local-stats
- stats
- mirrors
Last modified date
::::::::::::::::::
@ -81,12 +109,11 @@ synchronisation date is made available.
For PyPI, each mirror needs to maintain a URL with simple text content
that represents the last synchronisation date the mirror maintains.
The date is provided in GMT time, using the ISO 8601 format (see
http://en.wikipedia.org/wiki/ISO_8601).
The date is provided in GMT time, using the ISO 8601 format [#iso8601]_.
Each mirror will be responsible to maintain its last modified date.
Conventionally, this page should be reachable at: `/last-modified`.
This page must be located at : `/last-modified` and must be a
text/plain page.
Local statistics
::::::::::::::::
@ -124,42 +151,7 @@ They are then provided in a folder called `days`. For example:
- /local-stats/days/2008-11-07.bz2
- /local-stats/days/2008-11-08.bz2
Conventionally the name should be `local-stats`, but it can be any
name provided when the mirror is registered.
Statistics page
:::::::::::::::
PyPI and each mirror are responsible to provide the grand total page
at `/stats`. This page is calculated daily by PyPI, by reading all
mirrors' local stats and summing them.
Therefore the mirrors should not try to rebuild this stat page but
simply get the one on PyPI during each synchronization.
It has the same structure as `local-stats` but also provides counts
for months.
Examples:
- /stats/days/2008-11-06.bz2
- /stats/days/2008-11-07.bz2
- /stats/days/2008-11-08.bz2
- /stats/months/2008-11.bz2
- /stats/months/2008-10.bz2
Conventionally the name should be `stats`, but it can be any name
provided when the mirror is registered.
Mirrors listing page
::::::::::::::::::::
Like `/stats`, each mirror should get and provide a copy of the
`/mirrors` page.
Conventionally the name should be `mirrors`, but it can be any name
provided when the mirror is registered.
This page must be located at `/local-stats`.
How a mirror should synchronize with PyPI
@ -175,7 +167,7 @@ The mirroring protocol
XXX Need to describe the protocol here.
The zc.pypimirror package [#zcpkg]_ provides an application that
The z3c.pypimirror package [#zcpkg]_ provides an application that
respects this protocol to browse PyPI.
User-agent request header
@ -187,10 +179,9 @@ softwares.
This is also true for all clients like:
- `zc.buildout <http://pypi.python.org/pypi/zc.buildout>`_
- `setuptools <http://pypi.python.org/pypi/zc.buildout>`_
- `pip <http://pypi.python.org/pypi/zc.buildout>`_
- etc.
- zc.buildout [#zc.buildout]_.
- setuptools [#setuptools]_.
- pip [#pip]_.
XXX user agent registering mechanism at PyPI ?
@ -198,7 +189,13 @@ How a client can use PyPI and its mirrors
:::::::::::::::::::::::::::::::::::::::::
Clients that are browsing PyPI should be able to use alternative
mirrors, by reading the `/mirrors` page at PyPI.
mirrors, by getting the list of the mirrors using `mirrors.pypi.python.org`.
Code example::
>>> import socket
>>> socket.gethostbyname_ex('mirrors.pypi.python.org')[-1]
['82.94.164.163', '88.191.64.248']
The clients so far that could use this mechanism:
@ -212,9 +209,6 @@ Fail-over mechanism
Clients that are browsing PyPI should be able to use a fail-over
mechanism when PyPI or the used mirror is not responding.
This can be done by parsing the `/mirrors` page of PyPI or the one
located on any PyPI mirror.
It is up to the client to decide wich mirror should be used, maybe by
looking at its geographical location and its responsivness.
@ -242,9 +236,8 @@ compatible with any package index server out there.
Softwares that are compatible with PyPI and Distutils so far:
- `PloneSoftwareCenter <http://plone.org/products/plonesoftwarecenter>`_
wich is used to run plone.org products section.
- `EggBasket <http://www.chrisarndt.de/projects/eggbasket>`_
- PloneSoftwareCenter [#psc]_ wich is used to run plone.org products section.
- EggBasket [#eggbasket]_.
**An extra package index is not a mirror of PyPI, but can have some
mirrors itself.**
@ -257,8 +250,10 @@ indexes, it should be able to use each one of them as a potential
source of packages. Different indexes should be defined as a sorted
list for the client to look for a package.
Each independant index can of course provide a list of its mirrors, if
the `/mirrors` page is available.
Each independant index can of course provide a list of its mirrors.
XXX define how to get the hostname for the mirrors of an arbitrary
index.
That permits all combinations at client level, for a reliable
packaging system with all levels of privacy.
@ -275,6 +270,30 @@ References
.. [#zcpkg]
http://pypi.python.org/pypi/z3c.pypimirror
.. [#iso8601]
http://en.wikipedia.org/wiki/ISO_8601
.. [#zc.buildout]
http://pypi.python.org/pypi/zc.buildout
.. [#setuptools]
http://pypi.python.org/pypi/setuptools
.. [#pip]
http://pypi.python.org/pypi/pip
.. [#psc]
http://plone.org/products/plonesoftwarecenter
.. [#eggbasket]
http://www.chrisarndt.de/projects/eggbasket
Aknowledgments
==============
Martin von Loewis, Georg Brandl.
Copyright
=========