| rev |
line source |
|
jbe@37
|
1 seqlua: Extension for handling sequential data in Lua
|
|
jbe@37
|
2 =====================================================
|
|
jbe@0
|
3
|
|
jbe@54
|
4 This package is an extension for the Lua programming language (version 5.2)
|
|
jbe@54
|
5 which:
|
|
jbe@0
|
6
|
|
jbe@54
|
7 * allows ``ipairs(seq)`` to accept either tables or functions (i.e function
|
|
jbe@54
|
8 iterators) as an argument,
|
|
jbe@49
|
9 * adds a new function ``string.concat(separator, seq)`` that concats either
|
|
jbe@32
|
10 table entries or function return values,
|
|
jbe@49
|
11 * provides auxiliary C functions and macros to simplify iterating over both
|
|
jbe@38
|
12 tables and iterator functions with a generic statement.
|
|
jbe@0
|
13
|
|
jbe@33
|
14 Existing ``__ipairs`` or ``__index`` (but not ``__len``) metamethods are
|
|
jbe@33
|
15 respected by both the Lua functions and the C functions and macros. The
|
|
jbe@32
|
16 ``__ipairs`` metamethod takes precedence over ``__index``, while the
|
|
jbe@32
|
17 ``__len`` metamethod is never used.
|
|
jbe@32
|
18
|
|
jbe@37
|
19 Metamethod handling in detail is explained in the last section
|
|
jbe@37
|
20 ("Respected metamethods") at the bottom of this README.
|
|
jbe@37
|
21
|
|
jbe@49
|
22 In Lua, this extension is loaded by ``require "seqlua"``. In order to use the
|
|
jbe@49
|
23 auxiliary C functions and macros, add ``#include <seqlualib.h>`` to your C file
|
|
jbe@49
|
24 and ensure that the functions implemented in ``seqlualib.c`` are statically or
|
|
jbe@49
|
25 dynamically linked with your C Lua library.
|
|
jbe@49
|
26
|
|
jbe@37
|
27
|
|
jbe@37
|
28
|
|
jbe@37
|
29 Motivation
|
|
jbe@37
|
30 ----------
|
|
jbe@37
|
31
|
|
jbe@37
|
32 Sequential data (such as arrays or streams) is often represented in two
|
|
jbe@37
|
33 different ways:
|
|
jbe@37
|
34
|
|
jbe@37
|
35 * as an ordered set of values (usually implemented as an array in other
|
|
jbe@37
|
36 programming languages, or as a sequence in Lua: a table with numeric keys
|
|
jbe@37
|
37 {1..n} associated with a value each),
|
|
jbe@37
|
38 * as some sort of data stream (sometimes implemented as a class of objects
|
|
jbe@37
|
39 providing certain methods, or as an iterator function in Lua: a function that
|
|
jbe@37
|
40 returns the next value with every call, where nil indicates the end of the
|
|
jbe@37
|
41 stream).
|
|
jbe@37
|
42
|
|
jbe@37
|
43 Quite often, when functions work on sequential data, it shouldn't matter in
|
|
jbe@37
|
44 which form the sequential data is being provided to the function. As an
|
|
jbe@37
|
45 example, consider a function that is writing a sequence of strings to a file.
|
|
jbe@37
|
46 Such function could either be fed with an array of strings (a table with
|
|
jbe@37
|
47 numeric keys in Lua) or with a (possibly infinite) stream of data (an iterator
|
|
jbe@37
|
48 function in Lua).
|
|
jbe@37
|
49
|
|
jbe@37
|
50 A function in Lua that accepts a table, might look like as follows:
|
|
jbe@37
|
51
|
|
jbe@37
|
52 function write_lines(lines)
|
|
jbe@37
|
53 for i, line in ipairs(lines) do
|
|
jbe@37
|
54 io.stdout:write(line)
|
|
jbe@37
|
55 io.stdout:write("\n")
|
|
jbe@37
|
56 end
|
|
jbe@37
|
57 end
|
|
jbe@37
|
58
|
|
jbe@37
|
59 In contrast, a function in Lua that accepts an iterator function would have to
|
|
jbe@37
|
60 be implemented differently:
|
|
jbe@37
|
61
|
|
jbe@37
|
62 function write_lines(get_next_line)
|
|
jbe@37
|
63 for line in get_next_line do
|
|
jbe@37
|
64 io.stdout:write(line)
|
|
jbe@37
|
65 io.stdout:write("\n")
|
|
jbe@37
|
66 end
|
|
jbe@37
|
67 end
|
|
jbe@37
|
68
|
|
jbe@37
|
69 If one wanted to create a function that accepts either a sequence in form of a
|
|
jbe@37
|
70 table or an iterator function, then one might need to write:
|
|
jbe@37
|
71
|
|
jbe@41
|
72 do
|
|
jbe@41
|
73 local function write_line(line)
|
|
jbe@37
|
74 io.stdout:write(line)
|
|
jbe@37
|
75 io.stdout:write("\n")
|
|
jbe@37
|
76 end
|
|
jbe@41
|
77 function write_lines(lines)
|
|
jbe@41
|
78 if type(lines) == "function" then
|
|
jbe@41
|
79 for line in lines do
|
|
jbe@41
|
80 write_line(line)
|
|
jbe@41
|
81 end
|
|
jbe@41
|
82 else
|
|
jbe@41
|
83 for i, line in ipairs(lines) do
|
|
jbe@41
|
84 write_line(line)
|
|
jbe@41
|
85 end
|
|
jbe@41
|
86 end
|
|
jbe@41
|
87 end
|
|
jbe@37
|
88 end
|
|
jbe@37
|
89
|
|
jbe@41
|
90 Obviously, this isn't something we want to do in every function that accepts
|
|
jbe@37
|
91 sequential data. Therefore, we usually decide for one of the two first forms
|
|
jbe@48
|
92 and thus disallow the other possible representation of sequential data to be
|
|
jbe@48
|
93 passed to the function.
|
|
jbe@37
|
94
|
|
jbe@37
|
95 This extension, however, modifies Lua's ``ipairs`` statement in such way that
|
|
jbe@37
|
96 it automatically accepts either a table or an iterator function as argument.
|
|
jbe@54
|
97 Thus, the first of the three ``write_lines`` functions above will accept both
|
|
jbe@54
|
98 (table) sequences and (function) iterators.
|
|
jbe@37
|
99
|
|
jbe@37
|
100 In addition to the modification of ``ipairs``, it also provides C functions and
|
|
jbe@37
|
101 macros to iterate over values in the same manner as a generic loop statement
|
|
jbe@37
|
102 with ``ipairs`` would do.
|
|
jbe@37
|
103
|
|
jbe@37
|
104 Note that this extension doesn't aim to supersede Lua's concept of iterator
|
|
jbe@37
|
105 functions. While metamethods (see section "Respected metamethods" below) may be
|
|
jbe@37
|
106 used to customize iteration behavior on values, this extension isn't thought to
|
|
jbe@37
|
107 replace the common practice to use function closures as iterators. Consider the
|
|
jbe@37
|
108 following example:
|
|
jbe@37
|
109
|
|
jbe@37
|
110 local result = sql_query("SELECT * FROM actor ORDER BY birthdate")
|
|
jbe@37
|
111 write_lines(result:get_column_entries("name"))
|
|
jbe@37
|
112
|
|
jbe@37
|
113 The ``get_column_entries`` method can return a simple function closure that
|
|
jbe@37
|
114 returns the next entry in the "name" column (returning ``nil`` to indicate the
|
|
jbe@37
|
115 end). Such a closure can then be passed to another function that iterates
|
|
jbe@37
|
116 through a sequence of values by invoking ``ipairs`` with the general for-loop
|
|
jbe@37
|
117 (as previously shown).
|
|
jbe@37
|
118
|
|
jbe@37
|
119 Where desired, it is also possible to use metamethods to customize iteration
|
|
jbe@44
|
120 behavior:
|
|
jbe@44
|
121
|
|
jbe@44
|
122 function print_rows(rows)
|
|
jbe@44
|
123 for i, row in ipairs(rows) do
|
|
jbe@44
|
124 print_row(row)
|
|
jbe@44
|
125 end
|
|
jbe@44
|
126 end
|
|
jbe@44
|
127 local result = sql_query("SELECT * FROM actor ORDER BY birthday")
|
|
jbe@46
|
128 assert(type(result) == "userdata")
|
|
jbe@44
|
129
|
|
jbe@44
|
130 -- we may rely on the ``__index`` or ``__ipairs`` metamethod to
|
|
jbe@44
|
131 -- iterate through all result rows here:
|
|
jbe@44
|
132 print_rows(result) -- no need to use ":rows()" or a similar syntax
|
|
jbe@44
|
133
|
|
jbe@45
|
134 -- but we can also still pass an individual set of result rows to the
|
|
jbe@44
|
135 -- print_rows function:
|
|
jbe@44
|
136 print_rows{result[1], result[#result]}
|
|
jbe@44
|
137
|
|
jbe@44
|
138 This extension, however, doesn't respect the ``__len`` metamethod due to the
|
|
jbe@47
|
139 following considerations:
|
|
jbe@37
|
140
|
|
jbe@39
|
141 * An efficient implementation where ``for i, v in ipairs(tbl) do ... end`` does
|
|
jbe@39
|
142 neither create a closure nor repeatedly evaluate ``#tbl`` seems to be
|
|
jbe@39
|
143 impossible.
|
|
jbe@37
|
144 * Respecting ``__len`` could be used to implement sparse arrays, but this would
|
|
jbe@37
|
145 require iterating functions to expect ``nil`` as a potential value. This may
|
|
jbe@37
|
146 lead to problems because ``nil`` is usually also used to indicate the absence
|
|
jbe@37
|
147 of a value.
|
|
jbe@37
|
148
|
|
jbe@40
|
149 Though, if such behavior is desired, it can still be implemented through the
|
|
jbe@37
|
150 ``__ipairs`` metamethod.
|
|
jbe@37
|
151
|
|
jbe@48
|
152 Unless manually done by the user in the ``__ipairs`` metamethod, the ``ipairs``
|
|
jbe@48
|
153 function as well as the corresponding C functions and macros provided by this
|
|
jbe@48
|
154 extension never create any closures or other values that need to be garbage
|
|
jbe@48
|
155 collected.
|
|
jbe@37
|
156
|
|
jbe@0
|
157
|
|
jbe@0
|
158
|
|
jbe@0
|
159 Lua part of the library
|
|
jbe@0
|
160 -----------------------
|
|
jbe@0
|
161
|
|
jbe@30
|
162 The modified ``ipairs(seq)`` and the new ``string.concat(sep, seq)`` functions
|
|
jbe@30
|
163 accept either a table or a function as ``seq``. This is demonstrated in the
|
|
jbe@30
|
164 following examples:
|
|
jbe@0
|
165
|
|
jbe@0
|
166 require "seqlua"
|
|
jbe@0
|
167
|
|
jbe@0
|
168 t = {"a", "b", "c"}
|
|
jbe@0
|
169
|
|
jbe@54
|
170 for i, v in ipairs(t) do
|
|
jbe@0
|
171 print(i, v)
|
|
jbe@0
|
172 end
|
|
jbe@0
|
173 -- prints:
|
|
jbe@0
|
174 -- 1 a
|
|
jbe@0
|
175 -- 2 b
|
|
jbe@0
|
176 -- 3 c
|
|
jbe@0
|
177
|
|
jbe@25
|
178 print(string.concat(",", t))
|
|
jbe@25
|
179 -- prints: a,b,c
|
|
jbe@25
|
180
|
|
jbe@19
|
181 function alphabet()
|
|
jbe@0
|
182 local letter = nil
|
|
jbe@0
|
183 return function()
|
|
jbe@0
|
184 if letter == nil then
|
|
jbe@19
|
185 letter = "a"
|
|
jbe@19
|
186 elseif letter == "z" then
|
|
jbe@0
|
187 return nil
|
|
jbe@0
|
188 else
|
|
jbe@0
|
189 letter = string.char(string.byte(letter) + 1)
|
|
jbe@0
|
190 end
|
|
jbe@0
|
191 return letter
|
|
jbe@0
|
192 end
|
|
jbe@0
|
193 end
|
|
jbe@0
|
194
|
|
jbe@54
|
195 for i, v in ipairs(alphabet()) do
|
|
jbe@0
|
196 print(i, v)
|
|
jbe@0
|
197 end
|
|
jbe@0
|
198 -- prints:
|
|
jbe@0
|
199 -- 1 a
|
|
jbe@0
|
200 -- 2 b
|
|
jbe@0
|
201 -- 3 c
|
|
jbe@0
|
202 -- ...
|
|
jbe@0
|
203 -- 25 y
|
|
jbe@0
|
204 -- 26 z
|
|
jbe@0
|
205
|
|
jbe@25
|
206 print(string.concat(",", alphabet()))
|
|
jbe@25
|
207 -- prints: a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z
|
|
jbe@25
|
208
|
|
jbe@26
|
209 function filter(f)
|
|
jbe@26
|
210 return function(seq)
|
|
jbe@26
|
211 return coroutine.wrap(function()
|
|
jbe@54
|
212 for i, v in ipairs(seq) do f(v) end
|
|
jbe@26
|
213 end)
|
|
jbe@26
|
214 end
|
|
jbe@0
|
215 end
|
|
jbe@19
|
216
|
|
jbe@29
|
217 alpha_beta_x = filter(function(v)
|
|
jbe@28
|
218 if v == "a" then
|
|
jbe@28
|
219 coroutine.yield("alpha")
|
|
jbe@28
|
220 elseif v == "b" then
|
|
jbe@28
|
221 coroutine.yield("beta")
|
|
jbe@28
|
222 elseif type(v) == "number" then
|
|
jbe@23
|
223 for i = 1, v do
|
|
jbe@28
|
224 coroutine.yield("X")
|
|
jbe@23
|
225 end
|
|
jbe@0
|
226 end
|
|
jbe@26
|
227 end)
|
|
jbe@0
|
228
|
|
jbe@29
|
229 print((","):concat(alpha_beta_x{"a", 3, "b", "c", "d"}))
|
|
jbe@28
|
230 -- prints: alpha,X,X,X,beta
|
|
jbe@25
|
231
|
|
jbe@29
|
232 print((","):concat(alpha_beta_x(alphabet())))
|
|
jbe@28
|
233 -- prints: alpha,beta
|
|
jbe@27
|
234
|
|
jbe@0
|
235
|
|
jbe@37
|
236
|
|
jbe@0
|
237 C part of the library
|
|
jbe@0
|
238 ---------------------
|
|
jbe@0
|
239
|
|
jbe@0
|
240 In ``seqlualib.h``, the following macro is defined:
|
|
jbe@0
|
241
|
|
jbe@54
|
242 #define seqlua_iterloop(L, iter, idx) \
|
|
jbe@0
|
243 for ( \
|
|
jbe@54
|
244 seqlua_iterinit((L), (iter), (idx)); \
|
|
jbe@0
|
245 seqlua_iternext(iter); \
|
|
jbe@25
|
246 )
|
|
jbe@25
|
247
|
|
jbe@25
|
248 and
|
|
jbe@25
|
249
|
|
jbe@25
|
250 #define seqlua_iterloopauto(L, iter, idx) \
|
|
jbe@25
|
251 for ( \
|
|
jbe@54
|
252 seqlua_iterinit((L), (iter), (idx)); \
|
|
jbe@25
|
253 seqlua_iternext(iter); \
|
|
jbe@0
|
254 lua_pop((L), 1) \
|
|
jbe@0
|
255 )
|
|
jbe@0
|
256
|
|
jbe@23
|
257 This macro allows iteration over either tables or iterator functions as the
|
|
jbe@23
|
258 following example function demonstrates:
|
|
jbe@0
|
259
|
|
jbe@0
|
260 int printcsv(lua_State *L) {
|
|
jbe@0
|
261 seqlua_Iterator iter;
|
|
jbe@54
|
262 seqlua_iterloop(L, &iter, 1) {
|
|
jbe@0
|
263 if (seqlua_itercount(&iter) > 1) fputs(",", stdout);
|
|
jbe@0
|
264 fputs(luaL_tolstring(L, -1, NULL), stdout);
|
|
jbe@25
|
265 // two values need to be popped (the value pushed by
|
|
jbe@25
|
266 // seqlua_iternext and the value pushed by luaL_tolstring)
|
|
jbe@25
|
267 lua_pop(L, 2);
|
|
jbe@0
|
268 }
|
|
jbe@0
|
269 fputs("\n", stdout);
|
|
jbe@0
|
270 return 0;
|
|
jbe@0
|
271 }
|
|
jbe@0
|
272
|
|
jbe@11
|
273 printcsv{"a", "b", "c"}
|
|
jbe@11
|
274 -- prints: a,b,c
|
|
jbe@11
|
275
|
|
jbe@11
|
276 printcsv(assert(io.open("testfile")):lines())
|
|
jbe@11
|
277 -- prints: line1,line2,... of "testfile"
|
|
jbe@0
|
278
|
|
jbe@31
|
279 NOTE: During iteration using ``seqlua_iterloop``, ``seqlua_iterloopauto``, or
|
|
jbe@31
|
280 ``seqlua_iterinit``, three extra elements are stored on the stack (additionally
|
|
jbe@31
|
281 to the value). These extra elements are removed automatically when the loop ends
|
|
jbe@31
|
282 (i.e. when ``seqlua_iternext`` returns zero). The value pushed onto the stack
|
|
jbe@31
|
283 for every iteration step has to be removed manually from the stack, unless
|
|
jbe@31
|
284 ``seqlua_iterloopauto`` is used.
|
|
jbe@0
|
285
|
|
jbe@31
|
286
|
|
jbe@37
|
287
|
|
jbe@35
|
288 Respected metamethods
|
|
jbe@35
|
289 ---------------------
|
|
jbe@35
|
290
|
|
jbe@35
|
291 Regarding the behavior of the Lua functions and the C functions and macros
|
|
jbe@35
|
292 provided by this extension, an existing ``__index`` metamethod will be
|
|
jbe@35
|
293 respected automatically. An existing ``__ipairs`` metamethod, however, takes
|
|
jbe@35
|
294 precedence.
|
|
jbe@35
|
295
|
|
jbe@35
|
296 If the ``__ipairs`` field of a value's metatable is set, then it must always
|
|
jbe@35
|
297 refer to a function. When starting iteration over a value with such a
|
|
jbe@35
|
298 metamethod being set, then this function is called with ``self`` (i.e. the
|
|
jbe@35
|
299 value itself) passed as first argument. The return values of the ``__ipairs``
|
|
jbe@35
|
300 metamethod may take one of the following 4 forms:
|
|
jbe@35
|
301
|
|
jbe@35
|
302 * ``return function_or_callable, static_argument, startindex`` causes the three
|
|
jbe@35
|
303 arguments to be returned by ``ipairs`` without further modification. Using
|
|
jbe@35
|
304 the C macros and functions for iteration, the behavior is according to the
|
|
jbe@35
|
305 generic loop statement in Lua:
|
|
jbe@35
|
306 ``for i, v in function_or_callable, static_argument, startindex do ... end``
|
|
jbe@35
|
307 * ``return "raw", table`` will result in iteration over the table ``table``
|
|
jbe@35
|
308 using ``lua_rawgeti``
|
|
jbe@35
|
309 * ``return "index", table_or_userdata`` will result in iteration over the table
|
|
jbe@35
|
310 or userdata while respecting any ``__index`` metamethod of the table or
|
|
jbe@35
|
311 userdata value
|
|
jbe@35
|
312 * ``return "call", function_or_callable`` will use the callable value as
|
|
jbe@35
|
313 (function) iterator where the function is expected to return a single value
|
|
jbe@35
|
314 without any index (the index is inserted automatically when using the
|
|
jbe@35
|
315 ``ipairs`` function for iteration)
|
|
jbe@35
|
316
|
|
jbe@35
|
317 These possiblities are demonstrated by the following example code:
|
|
jbe@35
|
318
|
|
jbe@35
|
319 require "seqlua"
|
|
jbe@35
|
320
|
|
jbe@35
|
321 do
|
|
jbe@35
|
322 local function ipairsaux(t, i)
|
|
jbe@35
|
323 i = i + 1
|
|
jbe@35
|
324 if i <= 3 then
|
|
jbe@35
|
325 return i, t[i]
|
|
jbe@35
|
326 end
|
|
jbe@35
|
327 end
|
|
jbe@35
|
328 custom = setmetatable(
|
|
jbe@35
|
329 {"one", "two", "three", "four", "five"},
|
|
jbe@35
|
330 {
|
|
jbe@35
|
331 __ipairs = function(self)
|
|
jbe@35
|
332 return ipairsaux, self, 0
|
|
jbe@35
|
333 end
|
|
jbe@35
|
334 }
|
|
jbe@35
|
335 )
|
|
jbe@35
|
336 end
|
|
jbe@35
|
337 print(string.concat(",", custom))
|
|
jbe@36
|
338 -- prints: one,two,three
|
|
jbe@35
|
339 -- (note that "four" and "five" are not printed)
|
|
jbe@35
|
340
|
|
jbe@35
|
341 tbl = {"alpha", "beta"}
|
|
jbe@35
|
342
|
|
jbe@35
|
343 proxy1 = setmetatable({}, {__index = tbl})
|
|
jbe@35
|
344 for i, v in ipairs(proxy1) do print(i, v) end
|
|
jbe@35
|
345 -- prints:
|
|
jbe@35
|
346 -- 1 alpha
|
|
jbe@35
|
347 -- 2 beta
|
|
jbe@35
|
348
|
|
jbe@35
|
349 proxy2 = setmetatable({}, {
|
|
jbe@35
|
350 __ipairs = function(self)
|
|
jbe@35
|
351 return "index", proxy1
|
|
jbe@35
|
352 end
|
|
jbe@35
|
353 })
|
|
jbe@35
|
354 for i, v in ipairs(proxy2) do print(i, v) end
|
|
jbe@35
|
355 -- prints:
|
|
jbe@35
|
356 -- 1 alpha
|
|
jbe@35
|
357 -- 2 beta
|
|
jbe@35
|
358 print(proxy2[1])
|
|
jbe@35
|
359 -- prints: nil
|
|
jbe@35
|
360
|
|
jbe@35
|
361 cursor = setmetatable({
|
|
jbe@35
|
362 "alice", "bob", "charlie", pos=1
|
|
jbe@35
|
363 }, {
|
|
jbe@35
|
364 __call = function(self)
|
|
jbe@35
|
365 local value = self[self.pos]
|
|
jbe@35
|
366 if value == nil then
|
|
jbe@35
|
367 self.pos = 1
|
|
jbe@35
|
368 else
|
|
jbe@35
|
369 self.pos = self.pos + 1
|
|
jbe@35
|
370 end
|
|
jbe@35
|
371 return value
|
|
jbe@35
|
372 end,
|
|
jbe@35
|
373 __ipairs = function(self)
|
|
jbe@35
|
374 return "call", self
|
|
jbe@35
|
375 end
|
|
jbe@35
|
376 })
|
|
jbe@35
|
377 for i, v in ipairs(cursor) do print(i, v) end
|
|
jbe@35
|
378 -- prints:
|
|
jbe@35
|
379 -- 1 alice
|
|
jbe@35
|
380 -- 2 bob
|
|
jbe@35
|
381 -- 3 charlie
|
|
jbe@35
|
382 print(cursor())
|
|
jbe@35
|
383 -- prints: alice
|
|
jbe@35
|
384 for i, v in ipairs(cursor) do print(i, v) end
|
|
jbe@35
|
385 -- prints:
|
|
jbe@35
|
386 -- 1 bob
|
|
jbe@35
|
387 -- 2 charlie
|
|
jbe@35
|
388 -- (note that "alice" has been returned earlier)
|
|
jbe@35
|
389
|
|
jbe@35
|
390 coefficients = setmetatable({1.25, 3.14, 17.5}, {
|
|
jbe@35
|
391 __index = function(self) return 1 end,
|
|
jbe@35
|
392 __ipairs = function(self) return "raw", self end
|
|
jbe@35
|
393 })
|
|
jbe@35
|
394 for i, v in ipairs(coefficients) do print(i, v) end
|
|
jbe@35
|
395 -- prints:
|
|
jbe@35
|
396 -- 1 1.25
|
|
jbe@35
|
397 -- 2 3.14
|
|
jbe@35
|
398 -- 3 17.5
|
|
jbe@35
|
399 -- (note that iteration terminates even if coefficients[4] == 1)
|
|
jbe@35
|
400 print(coefficients[4])
|
|
jbe@35
|
401 -- prints: 1
|
|
jbe@35
|
402
|
|
jbe@35
|
403
|