python-peps/pep-3108.txt

PEP: 3108
Title: Standard Library Reorganization
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett@python.org>
Status: Draft
Type: Informational
Python-Version: 3.0
Content-Type: text/x-rst
Created: 01-Jan-2007
Post-History: 


Abstract
========

Just like the language itself, Python's standard library (stdlib) has
grown over the years to be very rich.  But over time some modules
have lost their need to be included with Python.  There has also been
an introduction of a naming convention for modules since Python's
inception that not all modules follow.

Python 3.0 has presented a chance to remove modules that do not have
long term usefulness.  This chance also allows for the renaming of
modules so that they follow the Python style guide [#pep-0008]_.  This
PEP lists modules that should not be included in Python 3.0 and what
modules need to be renamed.


Modules to Remove
=================

Guido pronounced that "silly old stuff" is to be deleted from the
stdlib for Py3K [#silly-old-stuff]_.  This is open-ended on purpose.
Each module to be removed needs to have a justification as to why it
should no longer be distributed with Python.  This can range from the
module being deprecated in Python 2.x to being for a platform that is
no longer widely used.

This section of the PEP lists the various modules to be removed. Each
subsection represents a different reason for modules to be
removed.  Each module must have a specific justification on top of
being listed in a specific subsection so as to make sure only modules
that truly deserve to be removed are in fact removed.

When a reason mentions how long it has been since a module has been
"uniquely edited", it is in reference to how long it has been since a
checkin was done specifically for the module and not for a change that
applied universally across the entire stdlib.  If an edit time is not
denoted as "unique" then it is the last time the file was edited,
period.


Previously deprecated
---------------------

PEP 4 lists all modules that have been deprecated in the stdlib
[#pep-0004]_.  All modules listed in the PEP at the time of the first
alpha release of Python 3.0 will be removed.

The entire contents of libold will also be removed.  These modules
have already been removed from being imported but are kept in the
distribution for Python for users that rely upon the code.

On top of the modules listed in PEP 4, other modules have been
documented as deprecated but have yet to be added to PEP 4.

==============  ========== 
Module          Deprecated 
==============  ========== 
buildtools      2.3        
cfmfile         2.4        
macfs           2.3        
md5             2.5        
sha             2.5        
strop           unknown
xmllib          unknown
==============  ========== 


Platform-specific with minimal use
----------------------------------

Python supports many platforms, some of which are not widely held.
And on some of these platforms there are modules that have limited use
to people on those platforms.  Because of their limited usefulness it
would be better to no longer burden the Python development team with
their maintenance.

* IRIX (which is no longer produced [#irix-retirement]_)

  + AL/al

    - Provides sound support on Indy and Indigo workstations.
    - Both workstations are no longer available.
    - Code has not been uniquely edited in three years.

  + cd

    - CD drive control for SGI systems.
    - SGI no longer sells machines with IRIX on them.
    - Code has not been uniquely edited in 14 years.

  + DEVICE/GL/gl/cgen/cgensuport

    - OpenGL access.
    - Has not been edited in at least eight years.
    - Third-party libraries provide better support (PyOpenGL [#pyopengl]_).

  + FL/fl/flp

    - Wrapper for the FORMS library [#irix-forms]_
    - FORMS has not been edited in 12 years.
    - Library is not widely used.
    - First eight hits on Google are for Python docs for fl.

  + fm

    - Wrapper to the IRIS Font Manager library.
    - Only available on SGI machines which no longer come with IRIX.

  + imgfile

    - Wrapper for SGI libimage library for imglib image files
      (``.rgb`` files).
    - Python Imaging Library provdes read-only support [#pil]_.
    - Not uniquely edited in 13 years.

  + jpeg

    - Wrapper for JPEG (de)compressor.
    - Code not uniquely edited in nine years.
    - Third-party libraries provide better support
      (Python Imaging Library [#pil]_).

* Solaris

  + SUNAUDIODEV/sunaudiodev

    - Access to the sound card on Sun machines.
    - Code not uniquely edited in over eight years.

* Mac

  + aetools/aepack/aetypes/MiniAEFrame

    - OSA support is better through third-party modules.
        
        * Appscript [#appscript]_.

    - Hard-coded endianness which breaks on Intel Macs.
    - Might need to rename if Carbon package dependent.

  + applesingle

    - Undocumented.
    - AppleSingle is a binary file format for A/UX.
    - A/UX no longer distributed.

  + autoGIL

    - Very bad model for using Python with the CFRunLoop.
    - Rename to _autogil if needed by Carbon package.

  + Carbon.CF
  
    - Half-heared attempt.

  + findertools

    - No longer useful.

  + FrameWork

    - Poorly documented.
    - Not updated to support Carbon Events.

    + icopen

      - Not needed on OS X.
      - Meant to replace 'open' which is usually a bad thing to do.

    + videoreader

      - No longer used.

    + W

      - No longer distributed with Python.


Hardly used
------------

Some modules that are platform-independent are hardly used.  This
can be from how easy it is to implement the functionality from scratch
or because the audience for the code is very small.

* audiodev

  + Undocumented.
  + Not edited in five years.
  + If removed sunaudio should go as well (also undocumented; not
    edited in over seven years).

* imputil

  + Undocumented.
  + Never updated to support absolute imports.

* mutex

  + Easy to implement using a semaphore and a queue.
  + Cannot block on a lock attempt.
  + Not uniquely edited since its addition 15 years ago.

* repr

  + Controls output of the repr of objects.
  + String slicing and string interpolation can do similar work.
  + Used by pdb, but do not need to expose API.

* stringold

  + Function versions of the methods on string objects.
  + Obsolete since Python 1.6.
  + Any functionality not in the string object or module will be moved
    to the string module (mostly constants).

* symtable/_symtable

  + Undocumented.

* toaiff

  + Undocumented.
  + Requires ``sox`` library to be installed on the system.

* user

  + Easily handled by allowing the application specify its own
    module name, check for existence, and import if found.

* new

  + Just a rebinding of names from the 'types' module.
  + Can also call ``type`` built-in to get most types easily.
  + Docstring states the module is no longer useful as of revision
    27241 (2002-06-15).

* pure

  + Written before Pure Atria was bought by Rational which was then
    bought by IBM (in other words, very old).


Obsolete
--------

Becoming obsolete signifies that either another module in the stdlib
or a widely distributed third-party library provides a better solution
for what the module is meant for.

* Bastion/rexec

  + Restricted execution / security.
  + Turned off in Python 2.3.
  + Modules deemed unsafe.

* commands/popen2

  + subprocess module replaces them [#pep-0324]_.

* dl

  + ctypes provides better support for same functionality.

* fpformat

  + All functionality is supported by string interpolation.

* getopt

  + optparse provides better functionality.
  + But ...

    - getopt is still widely used.
    - optparse does not handle the issue of when people set
      incompatible options very well.
    - optparse does not allow option arguments to be optional.

* ihooks

  + Undocumented.
  + For use with rexec which has been turned off since Python 2.3.

* imageop

  + Better support by third-party libraries
    (Python Imaging Library [#pil]_).

* linuxaudiodev

  + Replaced by ossaudiodev.

* stat

  + ``os.stat`` now returns a tuple with attributes.
  + Functions in the module should be made into methods for the object
    returned by os.stat.

* statvfs

  + ``os.statvfs`` now returns a tuple with attributes.

* thread

  + People should use 'threading' instead.

    - Rename 'thread' to _thread.
    - Deprecate dummy_thread and rename _dummy_thread.
    - Move thread.get_ident over to threading.

  + Guido has previously supported the deprecation
    [#thread-deprecation]_.


Modules to Rename
=================

Along with the stdlib gaining some modules that are no longer
relevant, there is also the issue of naming.  Many modules existed in
the stdlib before PEP 8 came into existence [#pep-0008]_.  This has
led to some naming inconsistencies that should be addressed.

Any module that has been suggested for removal and does not meet the
required naming scheme is *not* listed below.


PEP 8 violations
----------------

PEP 8 specifies that modules "should have short, all-lowercase names"
where "underscores can be used ... if it improves readability"
[#pep-0008]_.  The use of underscores is discouraged in package names.

==================  ==================================================
Current Name        Replacement Name
==================  ==================================================
_winreg             winreg (rename also because module has a public
                    interface and thus should not have a leading
                    underscore)
BaseHTTPServer      base_http_server
Carbon              carbon
CGIHTTPServer       cgi_http_server
ColorPicker         colorpicker
ConfigParser        configparser
Cookie              cookie
copy_reg            copyreg
DocXMLRPCServer     doc_xmlrpc_server
EasyDialogs         easydialogs
HTMLParser          htmlparser
MacOS               macos
Nav                 nav
PixMapWrapper       pixmap_wrapper
Queue               queue
repr                reprlib (rename because module name shadows a 
                    built-in)
ScrolledText        scrolledtext
SimpleHTTPServer    simple_http_server
SimpleXMLRPCServer  simple_xmlrpc_server
SocketServer        socketserver
StringIO            stringio
Tix                 tix
Tkinter             tkinter
UserDict            userdict
UserList            userlist
UserString          userstring
==================  ==================================================


Merging C and Python implementations of the same interface
----------------------------------------------------------

Several interfaces have both a Python and C implementation.  While it
is great to have a C implementation for speed with a Python
implementation as fallback, there is no need to expose the two
implementations independently in the stdlib.  For Python 3.0 all
interfaces with two implementations will be merged into a single
public interface.

The C module is to be given a leading underscore to delineate the fact
that it is not the reference implementation (the Python implementation
is).  This means that any semantic difference between the C and Python
versions must be dealt with before Python 3.0 or else the C
implementation will be removed until it can be fixed.

One interface that is not listed below is xml.etree.ElementTree.  This
is an externally maintained module and thus is not under the direct
control of the Python development team for renaming.  See `Open
Issues`_ for a discussion on this.

* pickle/cPickle

  + Rename cPickle to _pickle.
  + Semantic completeness of C implementation *not* verified.

* profile/cProfile

  + Rename cProfile to _profile.
  + Semantic completeness of C implementation *not* verified.

* StringIO/cStringIO

  + Rename StringIO to stringio.
  + Rename cStringIO to _stringio.
  + Semantic completeness of C implementation *not* verified.


No public, documented interface
-------------------------------

There are several modules in the stdlib that have no defined public
interface.  These modules exist as support code for other modules that
are exposed.  Because they are not meant to be used directly they
should be renamed to reflect this fact.

============  ===============================
Current Name  Replacement Name
============  ===============================
bdb           _bdb
cl            _cl
markupbase    _markupbase
opcode        _opcode
dummy_thread  _dummy_thread (assuming thread
              is deprecated)
============  ===============================


Transition Plan
===============

For modules to be removed
-------------------------

A PendingDeprecationWarning will be set in Python 2.6 for all modules
slated to be removed in Python 3.0.  This will allow people to know
which modules will not exist but without being overly noisy since
PendingDeprecationWarning is by default silenced.


For modules to be renamed
-------------------------

Modules will be renamed in Python 2.6.  The original names of the
modules will still work but will raise a PendingDeprecationWarning
upon import.  The refactoring tool for transitioning to Python 3.0
will refactor imports that use the new names.


Open Issues
===========

Consolidate dependent modules together into a single module or package?
-----------------------------------------------------------------------

The stdlib has several modules that have a level of dependency between
them (e.g., urllib and urllib2).  Usually one is a low-level module
that provides basic abilities with a corresponding higher-level API is
given in another module for simple use-cases.  In Python 3.0 we could
group these dependent modules together into a single module or package
to better reflect their relationship.

Keep in mind when looking at the groupings that deprecation or removal
is also a possibility if there is enough overlap or a module is
obsolete.

* Cookie/cookielib
* urllib/urllib2 (urlparse? httplib?)
* cgi/cgitb
* Tix/Tkinter
* getpass/pwd/spwd/grp
* mailbox/mhlib
* anydbm/whichdbm
* bsddb/dbhash
* pickle/pickletools
* HTMLParser/htmllib
* ftplib/netrc
* parser/symbol


Consolidate certain modules with similar themes together in a package?
----------------------------------------------------------------------

Packages are often used to group together modules that have a similar
theme but do not have any direct relationship or dependency upon each
other.  For Python 3.0 obvious groupings could be done since renaming
of various modules is already occurring.  Below is a list of possible
packages (pre-existing or new) that could be created.  In order to be
considered the package must contain three or more modules in order to
not overdue the introduction of new packages and to keep the stdlib
shallow.  No specific package name is proposed, just what modules
could go into the package.

* 'collections' package

  + heapq
  + Queue
  + UserDist (?)
  + UserList (?)

* Python implementations of built-in data types

  + UserDict
  + UserList
  + UserString

* Platform-specific

  + windows

    - _winreg
    - msilib
    - msvcrt
    - winsound

  + unix

    - crypt
    - dbm
    - fcntl
    - fpectl
    - gdbm
    - grp
    - nis
    - pipes
    - pwd
    - readline (?)
    - resource
    - spwd
    - syslog
    - termios (?)
    - tty (?)

  - mac (all Mac-specific modules)

* Profiling

  + cProfile
  + profile
  + hotshot
  + pstats

* 'email' package

  + imaplib
  + mailbox
  + mailcap
  + mhlib
  + poplib

* Databases

  + anydbm
  + dbhash
  + dbm
  + bsddb
  + dumbdbm
  + gdbm
  + sqlite3
  + whichdb

* Audio

  + aifc
  + audioop
  + chunk
  + ossaudiodev
  + sndhdr
  + sunau
  + wave
  + winsound

* XML-RPC

  + DocXMLRPCServer
  + SimpleXMLRPCServer
  + xmlrpclib

* Internet (leaving out all questionable modules leads to a Web
  grouping)

  + BaseHTTPServer
  + cgi
  + CGIHTTPServer
  + cgitb
  + Cookie
  + cookielib
  + ftplib (?)
  + htmlentitydefs
  + htmllib
  + HTMLParser
  + httplib
  + netrc (?)
  + nntplib (?)
  + sgmllib (?)
  + SimpleHTTPServer
  + stringprep
  + urllib
  + urllib2
  + urlparse
  + webbrowser
  + wsgiref

* Servers (each sub-grouping can be their own package)

  + HTTP

    - BaseHTTPServer
    - CGIHTTPServer
    - DocXMLRPCServer
    - SimpleHTTPServer
    - SimpleXMLRPCServer
    - wsgiref

  + Socket

    - asynchat
    - asyncore
    - SocketServer

* Bytecode

  + codeop
  + compiler
  + dis
  + py_compile
  + compileall

* Compression

  + bz2
  + gzip
  + zlib

* Date & time

  + calendar
  + datetime
  + time


Renaming of modules maintained outside of the stdlib
----------------------------------------------------

xml.etree.ElementTree not only does not meet PEP 8 naming standards
but it also has an exposed C implementation [#pep-0008]_.  It is an
externally maintained package, though [#pep-0360]_.  A request will be
made for the maintainer to change the name so that it matches PEP 8
and hides the C implementation.


Rejected Ideas
==============

Modules that were suggested for removal but were saved 
------------------------------------------------------

* asynchat/asyncore

  + Josiah Carlson has said he will help to maintain the modules.

* base64/quopri/uu

  + All still widely used.
  + 'codecs' module does not provide as nice of an API for basic
    usage.

* fileinput

  + Useful when having to work with stdin.

* nis

  + Testimonials from people that new installations of NIS are still
    occurring out in the world.

* telnetlib

  + Really handy for quick-and-dirty remote access.
  + Some hardware supports telnet for accessing it.


Introducing a new top-level package
-----------------------------------

It has been suggested that the entire stdlib be placed within its own
package.  This PEP will not address this issue as it has its own
design issues (naming, does it deserve special consideration in import
semantics, etc.).  Everything within this PEP can easily be handled if
a new top-level package is introduced.


References
==========

.. [#pep-0004] PEP 4: Deprecation of Standard Modules
    (http://www.python.org/dev/peps/pep-0004/)

.. [#pep-0008] PEP 8: Style Guide for Python Code
    (http://www.python.org/dev/peps/pep-0008/)

.. [#pep-0324] PEP 324: subprocess -- New process module
    (http://www.python.org/dev/peps/pep-0324/)

.. [#pep-0360] PEP 360: Externally Maintained Packages
    (http://www.python.org/dev/peps/pep-0360/)

.. [#module-index] Python Documentation: Global Module Index
    (http://docs.python.org/modindex.html)

.. [#timing-module] Python Library Reference: Obsolete
    (http://docs.python.org/lib/obsolete-modules.html)

.. [#silly-old-stuff] Python-Dev email: "Py3k release schedule worries"
    (http://mail.python.org/pipermail/python-3000/2006-December/005130.html)

.. [#thread-deprecation] Python-Dev email: Autoloading?
    (http://mail.python.org/pipermail/python-dev/2005-October/057244.html)

.. [#py-dev-summary-2004-11-01] Python-Dev Summary: 2004-11-01
    (http://www.python.org/dev/summary/2004-11-01_2004-11-15/#id10)

.. [#pyopengl] PyOpenGL
    (http://pyopengl.sourceforge.net/)

.. [#pil] Python Imaging Library (PIL)
    (http://www.pythonware.com/products/pil/)

.. [#twisted] Twisted
    (http://twistedmatrix.com/trac/)

.. [#irix-retirement] SGI Press Release:
    End of General Availability for MIPS IRIX Products -- December 2006
    (http://www.sgi.com/support/mips_irix.html)

.. [#irix-forms] FORMS Library by Mark Overmars
    (ftp://ftp.cs.ruu.nl/pub/SGI/FORMS)

.. [#sun-au] Wikipedia: Au file format
    (http://en.wikipedia.org/wiki/Au_file_format)

.. [#appscript] appscript
    (http://appscript.sourceforge.net/)


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: