Skip to content

Commit e33235a

Browse files
committed
refact(buf): also use SlidingWindowMapBuffer as optional context-manager
+ doc(tutorial): update use-cases + doc(changes): new bullet.
1 parent d0bd74e commit e33235a

File tree

5 files changed

+168
-165
lines changed

5 files changed

+168
-165
lines changed

doc/source/changes.rst

+8-6
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,18 @@ Changelog
55
2.1.0
66
======
77

8-
* **BREAKING API:** etrofit ``git.util.mman`` as context-manager,
8+
- **BREAKING API:** retrofit ``git.util.mman`` as context-manager,
99
to release memory-mapped regions held.
10-
11-
The *mmap-manager(s)* are re-entrant, but not thread-safe **context-manager(s)**,
12-
to be used within a ``with ...:`` block, ensuring any left-overs cursors are cleaned up.
13-
If not entered, :meth:`StaticWindowMapManager.make_cursor()` and/or
10+
11+
The *mmap-manager(s)* are re-entrant, but not thread-safe **context-manager(s)**,
12+
to be used within a ``with ...:`` block, ensuring any left-overs cursors are cleaned up.
13+
If not entered, :meth:`StaticWindowMapManager.make_cursor()` and/or
1414
:meth:`WindowCursor.use_region()` will scream.
1515

1616
Get them from ``smmap.managed_mmaps()``.
1717

18+
- Retrofit :class:`SlidingWindowMapBuffer` also as context-manager.
19+
1820
v0.9.0
1921
========
2022
- Fixed issue with resources never being freed as mmaps were never closed.
@@ -41,7 +43,7 @@ v0.8.1
4143
- A single bugfix
4244

4345

44-
v0.8.0
46+
v0.8.0
4547
========
4648

4749
- Initial Release

doc/source/tutorial.rst

+52-51
Original file line numberDiff line numberDiff line change
@@ -7,34 +7,34 @@ This text briefly introduces you to the basic design decisions and accompanying
77

88
Design
99
======
10-
Per application, there must be a *MemoryManager* to be used throughout the application.
10+
Per application, there must be a *MemoryManager* to be used throughout the application.
1111
It can be configured to keep your resources within certain limits.
1212

1313
To access mapped regions, you require a cursor. Cursors point to exactly one file and serve as handles into it.
1414
As long as it exists, the respective memory region will remain available.
1515

16-
For convenience, a buffer implementation is provided which handles cursors and resource allocation
16+
For convenience, a buffer implementation is provided which handles cursors and resource allocation
1717
behind its simple buffer like interface.
1818

1919

2020
Memory Managers
2121
================
22-
There are two types of memory managers, one uses *static* windows, the other one uses *sliding* windows.
23-
A window is a region of a file mapped into memory. Although the names might be somewhat misleading,
24-
as technically windows are always static, the *sliding* version will allocate relatively small windows
22+
There are two types of memory managers, one uses *static* windows, the other one uses *sliding* windows.
23+
A window is a region of a file mapped into memory. Although the names might be somewhat misleading,
24+
as technically windows are always static, the *sliding* version will allocate relatively small windows
2525
whereas the *static* version will always map the whole file.
2626

27-
The *static* memory-manager does nothing more than keeping a client count on the respective memory maps
28-
which always map the whole file, which allows to make some assumptions that can lead to simplified
27+
The *static* memory-manager does nothing more than keeping a client count on the respective memory maps
28+
which always map the whole file, which allows to make some assumptions that can lead to simplified
2929
data access and increased performance, but reduces the compatibility to 32 bit systems or giant files.
3030

31-
The *sliding* memory-manager therefore should be the default manager when preparing an application
31+
The *sliding* memory-manager therefore should be the default manager when preparing an application
3232
for handling huge amounts of data on 32 bit and 64 bit platforms
3333

3434
.. Note::
35-
The *mmap-manager(s)* are re-entrant, but not thread-safe **context-manager(s)**,
36-
to be used within a ``with ...:`` block, ensuring any left-overs cursors are cleaned up.
37-
If not entered, :meth:`StaticWindowMapManager.make_cursor()` and/or
35+
The *mmap-manager(s)* are re-entrant, but not thread-safe **context-manager(s)**,
36+
to be used within a ``with ...:`` block, ensuring any left-overs cursors are cleaned up.
37+
If not entered, :meth:`StaticWindowMapManager.make_cursor()` and/or
3838
:meth:`WindowCursor.use_region()` will scream.
3939

4040

@@ -44,7 +44,7 @@ Use the :math:`smmap.managed_mmaps()` to take care of all this::
4444
# This instance should be globally available in your application
4545
# It is configured to be well suitable for 32-bit or 64 bit applications.
4646
with smmap.managed_mmaps() as mman:
47-
47+
4848
# the manager provides much useful information about its current state
4949
# like the amount of open file handles or the amount of mapped memory
5050
mman.num_file_handles()
@@ -60,81 +60,82 @@ Cursors
6060

6161
with smmap.managed_mmaps() as mman:
6262
fc = smmap.test.lib.FileCreator(1024*1024*8, "test_file")
63-
63+
6464
# obtain a cursor to access some file.
6565
c = mman.make_cursor(fc.path)
66-
66+
6767
# the cursor is now associated with the file, but not yet usable
6868
assert c.is_associated()
6969
assert not c.is_valid()
70-
71-
# before you can use the cursor, you have to specify a window you want to
70+
71+
# before you can use the cursor, you have to specify a window you want to
7272
# access. The following just says you want as much data as possible starting
7373
# from offset 0.
7474
# To be sure your region could be mapped, query for validity
7575
assert c.use_region().is_valid() # use_region returns self
76-
76+
7777
# once a region was mapped, you must query its dimension regularly
7878
# to assure you don't try to access its buffer out of its bounds
7979
assert c.size()
8080
c.buffer()[0] # first byte
8181
c.buffer()[1:10] # first 9 bytes
8282
c.buffer()[c.size()-1] # last byte
83-
83+
8484
# its recommended not to create big slices when feeding the buffer
85-
# into consumers (e.g. struct or zlib).
85+
# into consumers (e.g. struct or zlib).
8686
# Instead, either give the buffer directly, or use pythons buffer command.
8787
buffer(c.buffer(), 1, 9) # first 9 bytes without copying them
88-
88+
8989
# you can query absolute offsets, and check whether an offset is included
9090
# in the cursor's data.
9191
assert c.ofs_begin() < c.ofs_end()
9292
assert c.includes_ofs(100)
93-
94-
# If you are over out of bounds with one of your region requests, the
93+
94+
# If you are over out of bounds with one of your region requests, the
9595
# cursor will be come invalid. It cannot be used in that state
9696
assert not c.use_region(fc.size, 100).is_valid()
9797
# map as much as possible after skipping the first 100 bytes
9898
assert c.use_region(100).is_valid()
99-
100-
# You can explicitly free cursor resources by unusing the cursor's region
99+
100+
# You must explicitly free cursor resources by unusing the cursor's region
101101
c.unuse_region()
102102
assert not c.is_valid()
103-
104103

105-
Now you would have to write your algorithms around this interface to properly slide through huge amounts of data.
106-
104+
105+
Now you would have to write your algorithms around this interface to properly slide through huge amounts of data.
106+
107107
Alternatively you can use a convenience interface.
108108

109109

110110
========
111111
Buffers
112112
========
113-
To make first use easier, at the expense of performance, there is a Buffer implementation which uses a cursor underneath.
113+
To make first use easier, at the expense of performance, there is a Buffer implementation
114+
which uses a cursor underneath.
114115

115-
With it, you can access all data in a possibly huge file without having to take care of setting the cursor to different regions yourself::
116+
With it, you can access all data in a possibly huge file
117+
without having to take care of setting the cursor to different regions yourself::
116118

117119
# Create a default buffer which can operate on the whole file
118-
buf = smmap.SlidingWindowMapBuffer(mman.make_cursor(fc.path))
119-
120-
# you can use it right away
121-
assert buf.cursor().is_valid()
122-
123-
buf[0] # access the first byte
124-
buf[-1] # access the last ten bytes on the file
125-
buf[-10:]# access the last ten bytes
126-
127-
# If you want to keep the instance between different accesses, use the
128-
# dedicated methods
129-
buf.end_access()
130-
assert not buf.cursor().is_valid() # you cannot use the buffer anymore
131-
assert buf.begin_access(offset=10) # start using the buffer at an offset
132-
133-
# it will stop using resources automatically once it goes out of scope
134-
135-
Disadvantages
136-
--------------
137-
Buffers cannot be used in place of strings or maps, hence you have to slice them to have valid
138-
input for the sorts of struct and zlib.
139-
A slice means a lot of data handling overhead which makes buffers slower compared to using cursors directly.
120+
with smmap.SlidingWindowMapBuffer(mman.make_cursor(fc.path)) as buf:
121+
122+
# you can use it right away
123+
assert buf.cursor().is_valid()
124+
125+
buf[0] # access the first byte
126+
buf[-1] # access the last ten bytes on the file
127+
buf[-10:]# access the last ten bytes
128+
129+
# If you want to keep the instance between different accesses, use the
130+
# dedicated methods
131+
buf.end_access()
132+
assert not buf.cursor().is_valid() # you cannot use the buffer anymore
133+
assert buf.begin_access(offset=10) # start using the buffer at an offset
134+
135+
136+
Disadvantages
137+
--------------
138+
Buffers cannot be used in place of strings or maps, hence you have to slice them to have valid
139+
input for the sorts of struct and zlib.
140+
A slice means a lot of data handling overhead which makes buffers slower compared to using cursors directly.
140141

smmap/buf.py

+8-3
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,14 @@ class SlidingWindowMapBuffer(object):
1818
The buffer is relative, that is if you map an offset, index 0 will map to the
1919
first byte at the offset you used during initialization or begin_access
2020
21-
**Note:** Although this type effectively hides the fact that there are mapped windows
22-
underneath, it can unfortunately not be used in any non-pure python method which
23-
needs a buffer or string"""
21+
.. Tip::
22+
Use it as a context-manager inside a ``with SlidingWindowMapBuffer(...):`` block.
23+
24+
.. Note::
25+
Although this type effectively hides the fact that there are mapped windows
26+
underneath, it can unfortunately not be used in any non-pure python method which
27+
needs a buffer or string
28+
"""
2429
__slots__ = (
2530
'_c', # our cursor
2631
'_size', # our supposed size

0 commit comments

Comments
 (0)