seqlua
view README @ 44:de5e7dd93863
Re-added example for customized iteration behavior
| author | jbe |
|---|---|
| date | Mon Aug 25 03:59:45 2014 +0200 (2014-08-25) |
| parents | 1a0ce77d9e4b |
| children | 35eb59c51be6 |
line source
1 seqlua: Extension for handling sequential data in Lua
2 =====================================================
4 This is an experimental package to extend Lua in the following manner:
6 * allow ``ipairs(seq)`` to accept either tables or functions (i.e function
7 iterators) as an argument,
8 * add a new function ``string.concat(separator, seq)`` that concats either
9 table entries or function return values,
10 * provide auxiliary C functions and macros to simplify iterating over both
11 tables and iterator functions with a generic statement.
13 Existing ``__ipairs`` or ``__index`` (but not ``__len``) metamethods are
14 respected by both the Lua functions and the C functions and macros. The
15 ``__ipairs`` metamethod takes precedence over ``__index``, while the
16 ``__len`` metamethod is never used.
18 Metamethod handling in detail is explained in the last section
19 ("Respected metamethods") at the bottom of this README.
23 Motivation
24 ----------
26 Sequential data (such as arrays or streams) is often represented in two
27 different ways:
29 * as an ordered set of values (usually implemented as an array in other
30 programming languages, or as a sequence in Lua: a table with numeric keys
31 {1..n} associated with a value each),
32 * as some sort of data stream (sometimes implemented as a class of objects
33 providing certain methods, or as an iterator function in Lua: a function that
34 returns the next value with every call, where nil indicates the end of the
35 stream).
37 Quite often, when functions work on sequential data, it shouldn't matter in
38 which form the sequential data is being provided to the function. As an
39 example, consider a function that is writing a sequence of strings to a file.
40 Such function could either be fed with an array of strings (a table with
41 numeric keys in Lua) or with a (possibly infinite) stream of data (an iterator
42 function in Lua).
44 A function in Lua that accepts a table, might look like as follows:
46 function write_lines(lines)
47 for i, line in ipairs(lines) do
48 io.stdout:write(line)
49 io.stdout:write("\n")
50 end
51 end
53 In contrast, a function in Lua that accepts an iterator function would have to
54 be implemented differently:
56 function write_lines(get_next_line)
57 for line in get_next_line do
58 io.stdout:write(line)
59 io.stdout:write("\n")
60 end
61 end
63 If one wanted to create a function that accepts either a sequence in form of a
64 table or an iterator function, then one might need to write:
66 do
67 local function write_line(line)
68 io.stdout:write(line)
69 io.stdout:write("\n")
70 end
71 function write_lines(lines)
72 if type(lines) == "function" then
73 for line in lines do
74 write_line(line)
75 end
76 else
77 for i, line in ipairs(lines) do
78 write_line(line)
79 end
80 end
81 end
82 end
84 Obviously, this isn't something we want to do in every function that accepts
85 sequential data. Therefore, we usually decide for one of the two first forms
86 and therefore disallow the other possible representation of sequential data to
87 be passed to the function.
89 This extension, however, modifies Lua's ``ipairs`` statement in such way that
90 it automatically accepts either a table or an iterator function as argument.
91 Thus, the first of the three ``write_lines`` functions above will accept both
92 (table) sequences and (function) iterators.
94 In addition to the modification of ``ipairs``, it also provides C functions and
95 macros to iterate over values in the same manner as a generic loop statement
96 with ``ipairs`` would do.
98 Note that this extension doesn't aim to supersede Lua's concept of iterator
99 functions. While metamethods (see section "Respected metamethods" below) may be
100 used to customize iteration behavior on values, this extension isn't thought to
101 replace the common practice to use function closures as iterators. Consider the
102 following example:
104 local result = sql_query("SELECT * FROM actor ORDER BY birthdate")
105 write_lines(result:get_column_entries("name"))
107 The ``get_column_entries`` method can return a simple function closure that
108 returns the next entry in the "name" column (returning ``nil`` to indicate the
109 end). Such a closure can then be passed to another function that iterates
110 through a sequence of values by invoking ``ipairs`` with the general for-loop
111 (as previously shown).
113 Where desired, it is also possible to use metamethods to customize iteration
114 behavior:
116 function print_rows(rows)
117 for i, row in ipairs(rows) do
118 print_row(row)
119 end
120 end
121 local result = sql_query("SELECT * FROM actor ORDER BY birthday")
123 -- we may rely on the ``__index`` or ``__ipairs`` metamethod to
124 -- iterate through all result rows here:
125 print_rows(result) -- no need to use ":rows()" or a similar syntax
127 -- but we can also still pass individual result rows to the
128 -- print_rows function:
129 print_rows{result[1], result[#result]}
131 This extension, however, doesn't respect the ``__len`` metamethod due to the
132 following reasons:
134 * An efficient implementation where ``for i, v in ipairs(tbl) do ... end`` does
135 neither create a closure nor repeatedly evaluate ``#tbl`` seems to be
136 impossible.
137 * Respecting ``__len`` could be used to implement sparse arrays, but this would
138 require iterating functions to expect ``nil`` as a potential value. This may
139 lead to problems because ``nil`` is usually also used to indicate the absence
140 of a value.
142 Though, if such behavior is desired, it can still be implemented through the
143 ``__ipairs`` metamethod.
145 Unless manually done by the user in the ``__ipairs`` metamethod, this extension
146 never creates any closures or other values that need to be garbage collected.
150 Lua part of the library
151 -----------------------
153 The modified ``ipairs(seq)`` and the new ``string.concat(sep, seq)`` functions
154 accept either a table or a function as ``seq``. This is demonstrated in the
155 following examples:
157 require "seqlua"
159 t = {"a", "b", "c"}
161 for i, v in ipairs(t) do
162 print(i, v)
163 end
164 -- prints:
165 -- 1 a
166 -- 2 b
167 -- 3 c
169 print(string.concat(",", t))
170 -- prints: a,b,c
172 function alphabet()
173 local letter = nil
174 return function()
175 if letter == nil then
176 letter = "a"
177 elseif letter == "z" then
178 return nil
179 else
180 letter = string.char(string.byte(letter) + 1)
181 end
182 return letter
183 end
184 end
186 for i, v in ipairs(alphabet()) do
187 print(i, v)
188 end
189 -- prints:
190 -- 1 a
191 -- 2 b
192 -- 3 c
193 -- ...
194 -- 25 y
195 -- 26 z
197 print(string.concat(",", alphabet()))
198 -- prints: a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z
200 function filter(f)
201 return function(seq)
202 return coroutine.wrap(function()
203 for i, v in ipairs(seq) do f(v) end
204 end)
205 end
206 end
208 alpha_beta_x = filter(function(v)
209 if v == "a" then
210 coroutine.yield("alpha")
211 elseif v == "b" then
212 coroutine.yield("beta")
213 elseif type(v) == "number" then
214 for i = 1, v do
215 coroutine.yield("X")
216 end
217 end
218 end)
220 print((","):concat(alpha_beta_x{"a", 3, "b", "c", "d"}))
221 -- prints: alpha,X,X,X,beta
223 print((","):concat(alpha_beta_x(alphabet())))
224 -- prints: alpha,beta
228 C part of the library
229 ---------------------
231 In ``seqlualib.h``, the following macro is defined:
233 #define seqlua_iterloop(L, iter, idx) \
234 for ( \
235 seqlua_iterinit((L), (iter), (idx)); \
236 seqlua_iternext(iter); \
237 )
239 and
241 #define seqlua_iterloopauto(L, iter, idx) \
242 for ( \
243 seqlua_iterinit((L), (iter), (idx)); \
244 seqlua_iternext(iter); \
245 lua_pop((L), 1) \
246 )
248 This macro allows iteration over either tables or iterator functions as the
249 following example function demonstrates:
251 int printcsv(lua_State *L) {
252 seqlua_Iterator iter;
253 seqlua_iterloop(L, &iter, 1) {
254 if (seqlua_itercount(&iter) > 1) fputs(",", stdout);
255 fputs(luaL_tolstring(L, -1, NULL), stdout);
256 // two values need to be popped (the value pushed by
257 // seqlua_iternext and the value pushed by luaL_tolstring)
258 lua_pop(L, 2);
259 }
260 fputs("\n", stdout);
261 return 0;
262 }
264 printcsv{"a", "b", "c"}
265 -- prints: a,b,c
267 printcsv(assert(io.open("testfile")):lines())
268 -- prints: line1,line2,... of "testfile"
270 NOTE: During iteration using ``seqlua_iterloop``, ``seqlua_iterloopauto``, or
271 ``seqlua_iterinit``, three extra elements are stored on the stack (additionally
272 to the value). These extra elements are removed automatically when the loop ends
273 (i.e. when ``seqlua_iternext`` returns zero). The value pushed onto the stack
274 for every iteration step has to be removed manually from the stack, unless
275 ``seqlua_iterloopauto`` is used.
279 Respected metamethods
280 ---------------------
282 Regarding the behavior of the Lua functions and the C functions and macros
283 provided by this extension, an existing ``__index`` metamethod will be
284 respected automatically. An existing ``__ipairs`` metamethod, however, takes
285 precedence.
287 If the ``__ipairs`` field of a value's metatable is set, then it must always
288 refer to a function. When starting iteration over a value with such a
289 metamethod being set, then this function is called with ``self`` (i.e. the
290 value itself) passed as first argument. The return values of the ``__ipairs``
291 metamethod may take one of the following 4 forms:
293 * ``return function_or_callable, static_argument, startindex`` causes the three
294 arguments to be returned by ``ipairs`` without further modification. Using
295 the C macros and functions for iteration, the behavior is according to the
296 generic loop statement in Lua:
297 ``for i, v in function_or_callable, static_argument, startindex do ... end``
298 * ``return "raw", table`` will result in iteration over the table ``table``
299 using ``lua_rawgeti``
300 * ``return "index", table_or_userdata`` will result in iteration over the table
301 or userdata while respecting any ``__index`` metamethod of the table or
302 userdata value
303 * ``return "call", function_or_callable`` will use the callable value as
304 (function) iterator where the function is expected to return a single value
305 without any index (the index is inserted automatically when using the
306 ``ipairs`` function for iteration)
308 These possiblities are demonstrated by the following example code:
310 require "seqlua"
312 do
313 local function ipairsaux(t, i)
314 i = i + 1
315 if i <= 3 then
316 return i, t[i]
317 end
318 end
319 custom = setmetatable(
320 {"one", "two", "three", "four", "five"},
321 {
322 __ipairs = function(self)
323 return ipairsaux, self, 0
324 end
325 }
326 )
327 end
328 print(string.concat(",", custom))
329 -- prints: one,two,three
330 -- (note that "four" and "five" are not printed)
332 tbl = {"alpha", "beta"}
334 proxy1 = setmetatable({}, {__index = tbl})
335 for i, v in ipairs(proxy1) do print(i, v) end
336 -- prints:
337 -- 1 alpha
338 -- 2 beta
340 proxy2 = setmetatable({}, {
341 __ipairs = function(self)
342 return "index", proxy1
343 end
344 })
345 for i, v in ipairs(proxy2) do print(i, v) end
346 -- prints:
347 -- 1 alpha
348 -- 2 beta
349 print(proxy2[1])
350 -- prints: nil
352 cursor = setmetatable({
353 "alice", "bob", "charlie", pos=1
354 }, {
355 __call = function(self)
356 local value = self[self.pos]
357 if value == nil then
358 self.pos = 1
359 else
360 self.pos = self.pos + 1
361 end
362 return value
363 end,
364 __ipairs = function(self)
365 return "call", self
366 end
367 })
368 for i, v in ipairs(cursor) do print(i, v) end
369 -- prints:
370 -- 1 alice
371 -- 2 bob
372 -- 3 charlie
373 print(cursor())
374 -- prints: alice
375 for i, v in ipairs(cursor) do print(i, v) end
376 -- prints:
377 -- 1 bob
378 -- 2 charlie
379 -- (note that "alice" has been returned earlier)
381 coefficients = setmetatable({1.25, 3.14, 17.5}, {
382 __index = function(self) return 1 end,
383 __ipairs = function(self) return "raw", self end
384 })
385 for i, v in ipairs(coefficients) do print(i, v) end
386 -- prints:
387 -- 1 1.25
388 -- 2 3.14
389 -- 3 17.5
390 -- (note that iteration terminates even if coefficients[4] == 1)
391 print(coefficients[4])
392 -- prints: 1