@@ -7,34 +7,34 @@ This text briefly introduces you to the basic design decisions and accompanying
7
7
8
8
Design
9
9
======
10
- Per application, there must be a *MemoryManager * to be used throughout the application.
10
+ Per application, there must be a *MemoryManager * to be used throughout the application.
11
11
It can be configured to keep your resources within certain limits.
12
12
13
13
To access mapped regions, you require a cursor. Cursors point to exactly one file and serve as handles into it.
14
14
As long as it exists, the respective memory region will remain available.
15
15
16
- For convenience, a buffer implementation is provided which handles cursors and resource allocation
16
+ For convenience, a buffer implementation is provided which handles cursors and resource allocation
17
17
behind its simple buffer like interface.
18
18
19
19
20
20
Memory Managers
21
21
================
22
- There are two types of memory managers, one uses *static * windows, the other one uses *sliding * windows.
23
- A window is a region of a file mapped into memory. Although the names might be somewhat misleading,
24
- as technically windows are always static, the *sliding * version will allocate relatively small windows
22
+ There are two types of memory managers, one uses *static * windows, the other one uses *sliding * windows.
23
+ A window is a region of a file mapped into memory. Although the names might be somewhat misleading,
24
+ as technically windows are always static, the *sliding * version will allocate relatively small windows
25
25
whereas the *static * version will always map the whole file.
26
26
27
- The *static * memory-manager does nothing more than keeping a client count on the respective memory maps
28
- which always map the whole file, which allows to make some assumptions that can lead to simplified
27
+ The *static * memory-manager does nothing more than keeping a client count on the respective memory maps
28
+ which always map the whole file, which allows to make some assumptions that can lead to simplified
29
29
data access and increased performance, but reduces the compatibility to 32 bit systems or giant files.
30
30
31
- The *sliding * memory-manager therefore should be the default manager when preparing an application
31
+ The *sliding * memory-manager therefore should be the default manager when preparing an application
32
32
for handling huge amounts of data on 32 bit and 64 bit platforms
33
33
34
34
.. Note ::
35
- The *mmap-manager(s) * are re-entrant, but not thread-safe **context-manager(s) **,
36
- to be used within a ``with ...: `` block, ensuring any left-overs cursors are cleaned up.
37
- If not entered, :meth: `StaticWindowMapManager.make_cursor() ` and/or
35
+ The *mmap-manager(s) * are re-entrant, but not thread-safe **context-manager(s) **,
36
+ to be used within a ``with ...: `` block, ensuring any left-overs cursors are cleaned up.
37
+ If not entered, :meth: `StaticWindowMapManager.make_cursor() ` and/or
38
38
:meth: `WindowCursor.use_region() ` will scream.
39
39
40
40
@@ -44,7 +44,7 @@ Use the :math:`smmap.managed_mmaps()` to take care of all this::
44
44
# This instance should be globally available in your application
45
45
# It is configured to be well suitable for 32-bit or 64 bit applications.
46
46
with smmap.managed_mmaps() as mman:
47
-
47
+
48
48
# the manager provides much useful information about its current state
49
49
# like the amount of open file handles or the amount of mapped memory
50
50
mman.num_file_handles()
@@ -60,81 +60,82 @@ Cursors
60
60
61
61
with smmap.managed_mmaps() as mman:
62
62
fc = smmap.test.lib.FileCreator(1024*1024*8, "test_file")
63
-
63
+
64
64
# obtain a cursor to access some file.
65
65
c = mman.make_cursor(fc.path)
66
-
66
+
67
67
# the cursor is now associated with the file, but not yet usable
68
68
assert c.is_associated()
69
69
assert not c.is_valid()
70
-
71
- # before you can use the cursor, you have to specify a window you want to
70
+
71
+ # before you can use the cursor, you have to specify a window you want to
72
72
# access. The following just says you want as much data as possible starting
73
73
# from offset 0.
74
74
# To be sure your region could be mapped, query for validity
75
75
assert c.use_region().is_valid() # use_region returns self
76
-
76
+
77
77
# once a region was mapped, you must query its dimension regularly
78
78
# to assure you don't try to access its buffer out of its bounds
79
79
assert c.size()
80
80
c.buffer()[0] # first byte
81
81
c.buffer()[1:10] # first 9 bytes
82
82
c.buffer()[c.size()-1] # last byte
83
-
83
+
84
84
# its recommended not to create big slices when feeding the buffer
85
- # into consumers (e.g. struct or zlib).
85
+ # into consumers (e.g. struct or zlib).
86
86
# Instead, either give the buffer directly, or use pythons buffer command.
87
87
buffer(c.buffer(), 1, 9) # first 9 bytes without copying them
88
-
88
+
89
89
# you can query absolute offsets, and check whether an offset is included
90
90
# in the cursor's data.
91
91
assert c.ofs_begin() < c.ofs_end()
92
92
assert c.includes_ofs(100)
93
-
94
- # If you are over out of bounds with one of your region requests, the
93
+
94
+ # If you are over out of bounds with one of your region requests, the
95
95
# cursor will be come invalid. It cannot be used in that state
96
96
assert not c.use_region(fc.size, 100).is_valid()
97
97
# map as much as possible after skipping the first 100 bytes
98
98
assert c.use_region(100).is_valid()
99
-
100
- # You can explicitly free cursor resources by unusing the cursor's region
99
+
100
+ # You must explicitly free cursor resources by unusing the cursor's region
101
101
c.unuse_region()
102
102
assert not c.is_valid()
103
-
104
103
105
- Now you would have to write your algorithms around this interface to properly slide through huge amounts of data.
106
-
104
+
105
+ Now you would have to write your algorithms around this interface to properly slide through huge amounts of data.
106
+
107
107
Alternatively you can use a convenience interface.
108
108
109
109
110
110
========
111
111
Buffers
112
112
========
113
- To make first use easier, at the expense of performance, there is a Buffer implementation which uses a cursor underneath.
113
+ To make first use easier, at the expense of performance, there is a Buffer implementation
114
+ which uses a cursor underneath.
114
115
115
- With it, you can access all data in a possibly huge file without having to take care of setting the cursor to different regions yourself::
116
+ With it, you can access all data in a possibly huge file
117
+ without having to take care of setting the cursor to different regions yourself::
116
118
117
119
# Create a default buffer which can operate on the whole file
118
- buf = smmap.SlidingWindowMapBuffer(mman.make_cursor(fc.path))
119
-
120
- # you can use it right away
121
- assert buf.cursor().is_valid()
122
-
123
- buf[0] # access the first byte
124
- buf[-1] # access the last ten bytes on the file
125
- buf[-10:]# access the last ten bytes
126
-
127
- # If you want to keep the instance between different accesses, use the
128
- # dedicated methods
129
- buf.end_access()
130
- assert not buf.cursor().is_valid() # you cannot use the buffer anymore
131
- assert buf.begin_access(offset=10) # start using the buffer at an offset
132
-
133
- # it will stop using resources automatically once it goes out of scope
134
-
135
- Disadvantages
136
- --------------
137
- Buffers cannot be used in place of strings or maps, hence you have to slice them to have valid
138
- input for the sorts of struct and zlib.
139
- A slice means a lot of data handling overhead which makes buffers slower compared to using cursors directly.
120
+ with smmap.SlidingWindowMapBuffer(mman.make_cursor(fc.path)) as buf:
121
+
122
+ # you can use it right away
123
+ assert buf.cursor().is_valid()
124
+
125
+ buf[0] # access the first byte
126
+ buf[-1] # access the last ten bytes on the file
127
+ buf[-10:]# access the last ten bytes
128
+
129
+ # If you want to keep the instance between different accesses, use the
130
+ # dedicated methods
131
+ buf.end_access()
132
+ assert not buf.cursor().is_valid() # you cannot use the buffer anymore
133
+ assert buf.begin_access(offset=10) # start using the buffer at an offset
134
+
135
+
136
+ Disadvantages
137
+ --------------
138
+ Buffers cannot be used in place of strings or maps, hence you have to slice them to have valid
139
+ input for the sorts of struct and zlib.
140
+ A slice means a lot of data handling overhead which makes buffers slower compared to using cursors directly.
140
141
0 commit comments