refact(buf): simplify API - no begin/end after construct

+ feat(mman): report missed exits.

Author: ankostis

doc/source/changes.rst

@@ -15,7 +15,9 @@ Changelog
 
   Get them from ``smmap.managed_mmaps()``.
 
-- Retrofit :class:`SlidingWindowMapBuffer` also as context-manager.
+- Simplify :class:`SlidingWindowMapBuffer` as create/close context-manager
+  (no ``begin_access()``, or ``end_access()``).
+
 
 v0.9.0
 ========

doc/source/tutorial.rst

@@ -8,13 +8,14 @@ This text briefly introduces you to the basic design decisions and accompanying
 Design
 ======
 Per application, there must be a *MemoryManager* to be used throughout the application.
-It can be configured to keep your resources within certain limits.
+It can be configured to keep your resources within certain limits (see :func:`smmap.managed_mmaps()`).
 
-To access mapped regions, you require a cursor. Cursors point to exactly one file and serve as handles into it.
+To access mapped regions, you require a cursor. Cursors point to exactly one file
+and serve as handles into it.
 As long as it exists, the respective memory region will remain available.
 
-For convenience, a buffer implementation is provided which handles cursors and resource allocation
-behind its simple buffer like interface.
+For convenience, a buffer implementation is provided (:class:`smmap.SlidingWindowMapBuffer`)
+which handles cursors and resource allocation behind its simple buffer like interface.
 
 
 Memory Managers
@@ -56,50 +57,48 @@ Cursors
 ========
 *Cursors* are handles that point onto a window, i.e. a region of a file mapped into memory. From them you may obtain a buffer through which the data of that window can actually be accessed::
 
-    import smmap.test.lib
-
-    with smmap.managed_mmaps() as mman:
-        fc = smmap.test.lib.FileCreator(1024*1024*8, "test_file")
+    import smmap.test.lib as tlib
 
+    with smmap.managed_mmaps() as mman, tlib.FileCreator(1024*1024*8, "test_file") as fc:
         # obtain a cursor to access some file.
-        c = mman.make_cursor(fc.path)
-
-        # the cursor is now associated with the file, but not yet usable
-        assert c.is_associated()
-        assert not c.is_valid()
-
-        # before you can use the cursor, you have to specify a window you want to
-        # access. The following just says you want as much data as possible starting
-        # from offset 0.
-        # To be sure your region could be mapped, query for validity
-        assert c.use_region().is_valid()		# use_region returns self
-
-        # once a region was mapped, you must query its dimension regularly
-        # to assure you don't try to access its buffer out of its bounds
-        assert c.size()
-        c.buffer()[0]			# first byte
-        c.buffer()[1:10]			# first 9 bytes
-        c.buffer()[c.size()-1] 	# last byte
-
-        # its recommended not to create big slices when feeding the buffer
-        # into consumers (e.g. struct or zlib).
-        # Instead, either give the buffer directly, or use pythons buffer command.
-        buffer(c.buffer(), 1, 9)	# first 9 bytes without copying them
-
-        # you can query absolute offsets, and check whether an offset is included
-        # in the cursor's data.
-        assert c.ofs_begin() < c.ofs_end()
-        assert c.includes_ofs(100)
-
-        # If you are over out of bounds with one of your region requests, the
-        # cursor will be come invalid. It cannot be used in that state
-        assert not c.use_region(fc.size, 100).is_valid()
-        # map as much as possible after skipping the first 100 bytes
-        assert c.use_region(100).is_valid()
-
-        # You must explicitly free cursor resources by unusing the cursor's region
-        c.unuse_region()
-        assert not c.is_valid()
+        with mman.make_cursor(fc.path) as c:
+
+            # the cursor is now associated with the file, but not yet usable
+            assert c.is_associated()
+            assert not c.is_valid()
+
+            # before you can use the cursor, you have to specify a window you want to
+            # access. The following just says you want as much data as possible starting
+            # from offset 0.
+            # To be sure your region could be mapped, query for validity
+            assert c.use_region().is_valid()		# use_region returns self
+
+            # once a region was mapped, you must query its dimension regularly
+            # to assure you don't try to access its buffer out of its bounds
+            assert c.size()
+            c.buffer()[0]			# first byte
+            c.buffer()[1:10]		# first 9 bytes
+            c.buffer()[c.size()-1] 	# last byte
+
+            # its recommended not to create big slices when feeding the buffer
+            # into consumers (e.g. struct or zlib).
+            # Instead, either give the buffer directly, or use pythons buffer command.
+            buffer(c.buffer(), 1, 9)	# first 9 bytes without copying them
+
+            # you can query absolute offsets, and check whether an offset is included
+            # in the cursor's data.
+            assert c.ofs_begin() < c.ofs_end()
+            assert c.includes_ofs(100)
+
+            # If you are over out of bounds with one of your region requests, the
+            # cursor will be come invalid. It cannot be used in that state
+            assert not c.use_region(fc.size, 100).is_valid()
+            # map as much as possible after skipping the first 100 bytes
+            assert c.use_region(100).is_valid()
+
+            # You must explicitly free cursor resources by unusing the cursor's region
+            c.unuse_region()
+            assert not c.is_valid()
 
 
 Now you would have to write your algorithms around this interface to properly slide through huge amounts of data.
@@ -116,21 +115,23 @@ which uses a cursor underneath.
 With it, you can access all data in a possibly huge file
 without having to take care of setting the cursor to different regions yourself::
 
-    # Create a default buffer which can operate on the whole file
-    with smmap.SlidingWindowMapBuffer(mman.make_cursor(fc.path)) as buf:
-
+    ## Create a default buffer which can operate on the whole file
+    cur = mman.make_cursor(fc.path)
+    with smmap.SlidingWindowMapBuffer(cur) as buf:
         # you can use it right away
         assert buf.cursor().is_valid()
 
         buf[0]	# access the first byte
         buf[-1]	# access the last ten bytes on the file
         buf[-10:]# access the last ten bytes
 
-        # If you want to keep the instance between different accesses, use the
-        # dedicated methods
-        buf.end_access()
-        assert not buf.cursor().is_valid()	# you cannot use the buffer anymore
-        assert buf.begin_access(offset=10)	# start using the buffer at an offset
+    ## You cannot use the buffer anymore.
+    assert not buf.cursor().is_valid()
+
+    ## If you want to keep the instance between different accesses,
+    # use another instance.
+    with smmap.SlidingWindowMapBuffer(cur, offset=10) as buf:
+        assert buf.cursor().is_valid()
 
 
 Disadvantages

smmap/buf.py

@@ -1,5 +1,6 @@
 """Module with a simple buffer implementation using the memory manager"""
 import sys
+import logging
 
 __all__ = ["SlidingWindowMapBuffer"]
 
@@ -10,6 +11,9 @@
     bytes = str  # @ReservedAssignment
 
 
+log = logging.getLogger(__name__)
+
+
 class SlidingWindowMapBuffer(object):
 
     """A buffer like object which allows direct byte-wise object and slicing into
@@ -29,11 +33,12 @@ class SlidingWindowMapBuffer(object):
     __slots__ = (
         '_c',           # our cursor
         '_size',        # our supposed size
+        '_entered',     # entry/exit accounting
     )
 
     def __init__(self, cursor=None, offset=0, size=sys.maxsize, flags=0):
-        """Initalize the instance to operate on the given cursor.
-        :param cursor: if not None, the associated cursor to the file you want to access
+        """Initialize the instance to operate on the given cursor.
+        :param cursor: The associated cursor to the file you want to access
             If None, you have call begin_access before using the buffer and provide a cursor
         :param offset: absolute offset in bytes
         :param size: the total size of the mapping. Defaults to the maximum possible size
@@ -43,24 +48,49 @@ def __init__(self, cursor=None, offset=0, size=sys.maxsize, flags=0):
             Hence it is in your own interest to provide a proper size !
         :param flags: Additional flags to be passed to os.open
         :raise ValueError: if the buffer could not achieve a valid state"""
+        if not cursor:
+            raise ValueError("Cursor cannot be null!")
         self._c = cursor
-        if cursor and not self.begin_access(cursor, offset, size, flags):
-            raise ValueError("Failed to allocate the buffer - probably the given offset is out of bounds")
-        # END handle offset
-
-    def __del__(self):
-        self.end_access()
+        self._entered = 0
+
+        if cursor.is_associated() and cursor.use_region(offset, size, flags).is_valid():
+            # if given size is too large or default, we computer a proper size
+            # If its smaller, we assume the combination between offset and size
+            # as chosen by the user is correct and use it !
+            # If not, the user is in trouble.
+            if size > cursor.file_size():
+                size = cursor.file_size() - offset
+            # END handle size
+            self._size = size
+        else:
+            raise ValueError("Cursor %s not associated or mapping region failed!" % cursor)
 
     def __enter__(self):
+        assert self._entered >= 0, self._entered
+        self._entered += 1
         return self
 
     def __exit__(self, exc_type, exc_value, traceback):
-        self.end_access()
+        assert self._entered >= 0, self._entered
+        self._entered -= 1
+        if self._entered == 0:
+            self.close()
+
+    def __del__(self):
+        if self._entered != 0:
+            log.warning("Missed %s exit(s) on %s!" % (self._entered, self))
+            self.close()
+
+    def _check_if_entered(self):
+        if self._entered <= 0:
+            raise ValueError('Context-manager %s not entered!' % self)
 
     def __len__(self):
         return self._size
 
     def __getitem__(self, i):
+        self._check_if_entered()
+
         if isinstance(i, slice):
             return self.__getslice__(i.start or 0, i.stop or self._size)
         c = self._c
@@ -73,6 +103,8 @@ def __getitem__(self, i):
         return c.buffer()[i - c.ofs_begin()]
 
     def __getslice__(self, i, j):
+        self._check_if_entered()
+
         c = self._c
         # fast path, slice fully included - safes a concatenate operation and
         # should be the default
@@ -124,44 +156,16 @@ def __getslice__(self, i, j):
         # END fast or slow path
     #{ Interface
 
-    def begin_access(self, cursor=None, offset=0, size=sys.maxsize, flags=0):
-        """Call this before the first use of this instance. The method was already
-        called by the constructor in case sufficient information was provided.
-
-        For more information no the parameters, see the __init__ method
-        :param path: if cursor is None the existing one will be used.
-        :return: True if the buffer can be used"""
-        if cursor:
-            self._c = cursor
-        # END update our cursor
-
-        # reuse existing cursors if possible
-        if self._c is not None and self._c.is_associated():
-            res = self._c.use_region(offset, size, flags).is_valid()
-            if res:
-                # if given size is too large or default, we computer a proper size
-                # If its smaller, we assume the combination between offset and size
-                # as chosen by the user is correct and use it !
-                # If not, the user is in trouble.
-                if size > self._c.file_size():
-                    size = self._c.file_size() - offset
-                # END handle size
-                self._size = size
-            # END set size
-            return res
-        # END use our cursor
-        return False
-
-    def end_access(self):
+    def close(self):
         """Call this method once you are done using the instance. It is automatically
         called on destruction, and should be called just in time to allow system
         resources to be freed.
 
-        Once you called end_access, you must call begin access before reusing this instance!"""
-        self._size = 0
-        if self._c is not None:
+        Once you called close, you must call begin access before reusing this instance!"""
+        if self._c:
             self._c.unuse_region()
-        # END unuse region
+            self._c = None
+            self._size = 0
 
     def cursor(self):
         """:return: the currently set cursor which provides access to the data"""

smmap/mman.py

@@ -359,6 +359,11 @@ def __exit__(self, exc_type, exc_value, traceback):
             if leaft_overs:
                 log.debug("Cleaned up %s left-over mmap-regions." % leaft_overs)
 
+    def __del__(self):
+        if self._entered != 0:
+            log.warning("Missed %s exit(s) on %s!" % (self._entered, self))
+            self.close()
+
     def close(self):
         self.collect()