From 419b7d99fb6d4c26ab5caee8a97de8c4ad73f325 Mon Sep 17 00:00:00 2001 From: Thomas Heller Date: Wed, 31 Jul 2002 18:48:36 +0000 Subject: [PATCH] The model exposed by the fixed buffer interface was changed: Retrieving a buffer from an object puts this in a locked state, and a releasebuffer function must be called to unlock the object again. Added releasefixedbuffer function slot, and renamed the get...fixedbuffer functions to acquire...fixedbuffer functions. Renamed the flag from Py_TPFLAG_HAVE_GETFIXEDBUFFER to Py_TPFLAG_HAVE_FIXEDBUFFER. (Is the 'fixed buffer' name still useful, or should we use 'static buffer' instead?) Added posting date (was posted to c.l.p and python-dev). --- pep-0298.txt | 115 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 67 insertions(+), 48 deletions(-) diff --git a/pep-0298.txt b/pep-0298.txt index 32f067958..898cc9290 100644 --- a/pep-0298.txt +++ b/pep-0298.txt @@ -7,7 +7,7 @@ Status: Draft Type: Standards Track Created: 26-Jul-2002 Python-Version: 2.3 -Post-History: +Post-History: 30-Jul-2002 Abstract @@ -16,15 +16,20 @@ Abstract 'fixed buffer interface'. The fixed buffer interface fixes the flaws of the 'old' buffer - interface as defined in Python versions up to and including 2.2, - see [1]: + interface [1] as defined in Python versions up to and including + 2.2, and has the following semantics: - The lifetime of the retrieved pointer is clearly defined. + The lifetime of the retrieved pointer is clearly defined and + controlled by the client. The buffer size is returned as a 'size_t' data type, which allows access to large buffers on platforms where sizeof(int) != sizeof(void *). + (Guido comments: This second sounds like a change we could also + make to the "old" buffer interface, if we introduce another flag + bit that's *not* part of the default flags.) + Specification @@ -32,10 +37,16 @@ Specification size and the pointer to the internal memory block of any python object which chooses to implement this interface. - The size and pointer returned must be valid as long as the object - is alive (has a positive reference count). So, only objects which - never reallocate or resize the memory block are allowed to - implement this interface. + Retrieving a buffer from an object puts this object in a locked + state during which the buffer may not be freed, resized, or + reallocated. + + The object must be unlocked again by releasing the buffer if it's + no longer used by calling another function in the fixed buffer + interface. If the object never resizes or reallocates the buffer + during it's lifetime, this function may be NULL. Failure to call + this function (if it is != NULL) is a programming error and may + have unexpected results. The fixed buffer interface omits the memory segment model which is present in the old buffer interface - only a single memory block @@ -46,16 +57,16 @@ Implementation Define a new flag in Include/object.h: - /* PyBufferProcs contains bf_getfixedreadbuffer - and bf_getfixedwritebuffer */ - #define Py_TPFLAGS_HAVE_GETFIXEDBUFFER (1L<<15) + /* PyBufferProcs contains bf_acquirefixedreadbuffer, + bf_acquirefixedwritebuffer, and bf_releasefixedbuffer */ + #define Py_TPFLAGS_HAVE_FIXEDBUFFER (1L<<15) This flag would be included in Py_TPFLAGS_DEFAULT: #define Py_TPFLAGS_DEFAULT ( \ .... - Py_TPFLAGS_HAVE_GETFIXEDBUFFER | \ + Py_TPFLAGS_HAVE_FIXEDBUFFER | \ .... 0) @@ -63,8 +74,11 @@ Implementation Extend the PyBufferProcs structure by new fields in Include/object.h: - typedef size_t (*getfixedreadbufferproc)(PyObject *, void **); - typedef size_t (*getfixedwritebufferproc)(PyObject *, void **); + typedef size_t (*acquirefixedreadbufferproc)(PyObject *, + const void **); + typedef size_t (*acquirefixedwritebufferproc)(PyObject *, + void **); + typedef void (*releasefixedbufferproc)(PyObject *); typedef struct { getreadbufferproc bf_getreadbuffer; @@ -72,42 +86,51 @@ Implementation getsegcountproc bf_getsegcount; getcharbufferproc bf_getcharbuffer; /* fixed buffer interface functions */ - getfixedreadbufferproc bf_getfixedreadbufferproc; - getfixedwritebufferproc bf_getfixedwritebufferproc; + acquirefixedreadbufferproc bf_acquirefixedreadbuffer; + acquirefixedwritebufferproc bf_acquirefixedwritebuffer; + releasefixedbufferproc bf_releasefixedbuffer; } PyBufferProcs; - The new fields are present if the Py_TPFLAGS_HAVE_GETFIXEDBUFFER + The new fields are present if the Py_TPFLAGS_HAVE_FIXEDBUFFER flag is set in the object's type. - The Py_TPFLAGS_HAVE_GETFIXEDBUFFER flag implies the + The Py_TPFLAGS_HAVE_FIXEDBUFFER flag implies the Py_TPFLAGS_HAVE_GETCHARBUFFER flag. - The getfixedreadbufferproc and getfixedwritebufferproc functions - return the size in bytes of the memory block on success, and fill - in the passed void * pointer on success. If these functions fail - - either because an error occurs or no memory block is exposed - - they must set the void * pointer to NULL and raise an exception. - The return value is undefined in these cases and should not be - used. + The acquirefixedreadbufferproc and acquirefixedwritebufferproc + functions return the size in bytes of the memory block on success, + and fill in the passed void * pointer on success. If these + functions fail - either because an error occurs or no memory block + is exposed - they must set the void * pointer to NULL and raise an + exception. The return value is undefined in these cases and + should not be used. - Usually the getfixedwritebufferproc and getfixedreadbufferproc - functions aren't called directly, they are called through - convenience functions declared in Include/abstract.h: + If calls to these functions succeed, eventually the buffer must be + released by a call to the releasefixedbufferproc, supplying the + original object as argument. The releasefixedbufferproc cannot + fail. - int PyObject_AsFixedReadBuffer(PyObject *obj, - void **buffer, - size_t *buffer_len); + Usually these functions aren't called directly, they are called + through convenience functions declared in Include/abstract.h: - int PyObject_AsFixedWriteBuffer(PyObject *obj, - void **buffer, - size_t *buffer_len); + int PyObject_AquireFixedReadBuffer(PyObject *obj, + const void **buffer, + size_t *buffer_len); - These functions return 0 on success, set buffer to the memory - location and buffer_len to the length of the memory block in - bytes. On failure, or if the fixed buffer interface is not + int PyObject_AcquireFixedWriteBuffer(PyObject *obj, + void **buffer, + size_t *buffer_len); + + void PyObject_ReleaseFixedBuffer(PyObject *obj); + + The former two functions return 0 on success, set buffer to the + memory location and buffer_len to the length of the memory block + in bytes. On failure, or if the fixed buffer interface is not implemented by obj, they return -1 and set an exception. + The latter function doesn't return anything, and cannot fail. + Backward Compatibility @@ -123,10 +146,8 @@ Reference Implementation Additional Notes/Comments - Python strings, Unicode strings, mmap objects, and maybe other - types would expose the fixed buffer interface, but the array type - would *not*, because its memory block may be reallocated during - its lifetime. + Python strings, Unicode strings, mmap objects, and array objects + would expose the fixed buffer interface. Community Feedback @@ -137,13 +158,11 @@ Community Feedback because even innocent looking calls to the Python API like Py_DECREF() may trigger execution of arbitrary Python code. - Neil Hodgson wants to expose pointers to memory blocks with - limited lifetime: do some kind of lock operation on the object, - retrieve the pointer, use it, and unlock the object again. While - the author sees the need for this, it cannot be addressed by this - proposal. Beeing required to call a function after not using the - pointer received by the getfixedbufferprocs any more seems too - error prone. + The first version of this proposal didn't have the release + function, but it turned out that this would have been too + restrictive: mmap and array objects wouldn't have been able to + implement it, because mmap objects can be closed anytime if not + locked, and array objects could resize or reallocate the buffer. Credits