seqlua
view README @ 37:adf671e3458c
Added section "Motivation" to README
| author | jbe |
|---|---|
| date | Mon Aug 25 02:52:47 2014 +0200 (2014-08-25) |
| parents | 4f82d3b3dbd9 |
| children | acc53a77fdc9 |
line source
1 seqlua: Extension for handling sequential data in Lua
2 =====================================================
4 This is an experimental package to extend Lua in the following manner:
6 * allow ``ipairs(seq)`` to accept tables as well as functions (i.e function
7 iterators),
8 * add a new function ``string.concat(separator, seq)`` to concat either
9 table entries or function return values,
10 * provide auxiliary C functions and macros to simplify iterating over both
11 tables and iterator functions with the same statement.
13 Existing ``__ipairs`` or ``__index`` (but not ``__len``) metamethods are
14 respected by both the Lua functions and the C functions and macros. The
15 ``__ipairs`` metamethod takes precedence over ``__index``, while the
16 ``__len`` metamethod is never used.
18 Metamethod handling in detail is explained in the last section
19 ("Respected metamethods") at the bottom of this README.
23 Motivation
24 ----------
26 Sequential data (such as arrays or streams) is often represented in two
27 different ways:
29 * as an ordered set of values (usually implemented as an array in other
30 programming languages, or as a sequence in Lua: a table with numeric keys
31 {1..n} associated with a value each),
32 * as some sort of data stream (sometimes implemented as a class of objects
33 providing certain methods, or as an iterator function in Lua: a function that
34 returns the next value with every call, where nil indicates the end of the
35 stream).
37 Quite often, when functions work on sequential data, it shouldn't matter in
38 which form the sequential data is being provided to the function. As an
39 example, consider a function that is writing a sequence of strings to a file.
40 Such function could either be fed with an array of strings (a table with
41 numeric keys in Lua) or with a (possibly infinite) stream of data (an iterator
42 function in Lua).
44 A function in Lua that accepts a table, might look like as follows:
46 function write_lines(lines)
47 for i, line in ipairs(lines) do
48 io.stdout:write(line)
49 io.stdout:write("\n")
50 end
51 end
53 In contrast, a function in Lua that accepts an iterator function would have to
54 be implemented differently:
56 function write_lines(get_next_line)
57 for line in get_next_line do
58 io.stdout:write(line)
59 io.stdout:write("\n")
60 end
61 end
63 If one wanted to create a function that accepts either a sequence in form of a
64 table or an iterator function, then one might need to write:
66 function write_lines(lines)
67 local iter1, iter2, iter3
68 if type(lines) == "function" then
69 iter1 = lines
70 else
71 iter1, iter2, iter3 = ipairs(lines)
72 end
73 for i, line in iter1, iter2, iter3 do
74 io.stdout:write(line)
75 io.stdout:write("\n")
76 end
77 end
79 Obviously, this isn't something we want to write in every function that accepts
80 sequential data. Therefore, we usually decide for one of the two first forms
81 and therefore disallow the other possible representation of sequential data to
82 be passed to the function.
84 This extension, however, modifies Lua's ``ipairs`` statement in such way that
85 it automatically accepts either a table or an iterator function as argument.
86 Thus, the first of the three functions above will accept both (table) sequences
87 and (function) iterators.
89 In addition to the modification of ``ipairs``, it also provides C functions and
90 macros to iterate over values in the same manner as a generic loop statement
91 with ``ipairs`` would do.
93 Note that this extension doesn't aim to supersede Lua's concept of iterator
94 functions. While metamethods (see section "Respected metamethods" below) may be
95 used to customize iteration behavior on values, this extension isn't thought to
96 replace the common practice to use function closures as iterators. Consider the
97 following example:
99 local result = sql_query("SELECT * FROM actor ORDER BY birthdate")
100 write_lines(result:get_column_entries("name"))
102 The ``get_column_entries`` method can return a simple function closure that
103 returns the next entry in the "name" column (returning ``nil`` to indicate the
104 end). Such a closure can then be passed to another function that iterates
105 through a sequence of values by invoking ``ipairs`` with the general for-loop
106 (as previously shown).
108 Where desired, it is also possible to use metamethods to customize iteration
109 behavior. Consider:
111 function process_row(database_row)
112 -- some function
113 end
114 local result = sql_query("SELECT * FROM actor")
115 process_row(result) -- without writing :rows() or similar
117 This extension, however, doesn't respect the ``__len`` metamethod due to the
118 following reasons:
120 * An efficient implementation seems to be impossible, where
121 ``for i, v in ipairs(tbl) do ... end`` does neither create a closure
122 nor repeatedly evaluate ``#tbl``.
123 * Respecting ``__len`` could be used to implement sparse arrays, but this would
124 require iterating functions to expect ``nil`` as a potential value. This may
125 lead to problems because ``nil`` is usually also used to indicate the absence
126 of a value.
128 If such behavior is desired, though, it can still be implemented through the
129 ``__ipairs`` metamethod.
131 Unless manually done by the user in the ``__ipairs`` metamethod, this extension
132 never creates any closures or other values that need to be garbage collected.
136 Lua part of the library
137 -----------------------
139 The modified ``ipairs(seq)`` and the new ``string.concat(sep, seq)`` functions
140 accept either a table or a function as ``seq``. This is demonstrated in the
141 following examples:
143 require "seqlua"
145 t = {"a", "b", "c"}
147 for i, v in ipairs(t) do
148 print(i, v)
149 end
150 -- prints:
151 -- 1 a
152 -- 2 b
153 -- 3 c
155 print(string.concat(",", t))
156 -- prints: a,b,c
158 function alphabet()
159 local letter = nil
160 return function()
161 if letter == nil then
162 letter = "a"
163 elseif letter == "z" then
164 return nil
165 else
166 letter = string.char(string.byte(letter) + 1)
167 end
168 return letter
169 end
170 end
172 for i, v in ipairs(alphabet()) do
173 print(i, v)
174 end
175 -- prints:
176 -- 1 a
177 -- 2 b
178 -- 3 c
179 -- ...
180 -- 25 y
181 -- 26 z
183 print(string.concat(",", alphabet()))
184 -- prints: a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z
186 function filter(f)
187 return function(seq)
188 return coroutine.wrap(function()
189 for i, v in ipairs(seq) do f(v) end
190 end)
191 end
192 end
194 alpha_beta_x = filter(function(v)
195 if v == "a" then
196 coroutine.yield("alpha")
197 elseif v == "b" then
198 coroutine.yield("beta")
199 elseif type(v) == "number" then
200 for i = 1, v do
201 coroutine.yield("X")
202 end
203 end
204 end)
206 print((","):concat(alpha_beta_x{"a", 3, "b", "c", "d"}))
207 -- prints: alpha,X,X,X,beta
209 print((","):concat(alpha_beta_x(alphabet())))
210 -- prints: alpha,beta
214 C part of the library
215 ---------------------
217 In ``seqlualib.h``, the following macro is defined:
219 #define seqlua_iterloop(L, iter, idx) \
220 for ( \
221 seqlua_iterinit((L), (iter), (idx)); \
222 seqlua_iternext(iter); \
223 )
225 and
227 #define seqlua_iterloopauto(L, iter, idx) \
228 for ( \
229 seqlua_iterinit((L), (iter), (idx)); \
230 seqlua_iternext(iter); \
231 lua_pop((L), 1) \
232 )
234 This macro allows iteration over either tables or iterator functions as the
235 following example function demonstrates:
237 int printcsv(lua_State *L) {
238 seqlua_Iterator iter;
239 seqlua_iterloop(L, &iter, 1) {
240 if (seqlua_itercount(&iter) > 1) fputs(",", stdout);
241 fputs(luaL_tolstring(L, -1, NULL), stdout);
242 // two values need to be popped (the value pushed by
243 // seqlua_iternext and the value pushed by luaL_tolstring)
244 lua_pop(L, 2);
245 }
246 fputs("\n", stdout);
247 return 0;
248 }
250 printcsv{"a", "b", "c"}
251 -- prints: a,b,c
253 printcsv(assert(io.open("testfile")):lines())
254 -- prints: line1,line2,... of "testfile"
256 NOTE: During iteration using ``seqlua_iterloop``, ``seqlua_iterloopauto``, or
257 ``seqlua_iterinit``, three extra elements are stored on the stack (additionally
258 to the value). These extra elements are removed automatically when the loop ends
259 (i.e. when ``seqlua_iternext`` returns zero). The value pushed onto the stack
260 for every iteration step has to be removed manually from the stack, unless
261 ``seqlua_iterloopauto`` is used.
265 Respected metamethods
266 ---------------------
268 Regarding the behavior of the Lua functions and the C functions and macros
269 provided by this extension, an existing ``__index`` metamethod will be
270 respected automatically. An existing ``__ipairs`` metamethod, however, takes
271 precedence.
273 If the ``__ipairs`` field of a value's metatable is set, then it must always
274 refer to a function. When starting iteration over a value with such a
275 metamethod being set, then this function is called with ``self`` (i.e. the
276 value itself) passed as first argument. The return values of the ``__ipairs``
277 metamethod may take one of the following 4 forms:
279 * ``return function_or_callable, static_argument, startindex`` causes the three
280 arguments to be returned by ``ipairs`` without further modification. Using
281 the C macros and functions for iteration, the behavior is according to the
282 generic loop statement in Lua:
283 ``for i, v in function_or_callable, static_argument, startindex do ... end``
284 * ``return "raw", table`` will result in iteration over the table ``table``
285 using ``lua_rawgeti``
286 * ``return "index", table_or_userdata`` will result in iteration over the table
287 or userdata while respecting any ``__index`` metamethod of the table or
288 userdata value
289 * ``return "call", function_or_callable`` will use the callable value as
290 (function) iterator where the function is expected to return a single value
291 without any index (the index is inserted automatically when using the
292 ``ipairs`` function for iteration)
294 These possiblities are demonstrated by the following example code:
296 require "seqlua"
298 do
299 local function ipairsaux(t, i)
300 i = i + 1
301 if i <= 3 then
302 return i, t[i]
303 end
304 end
305 custom = setmetatable(
306 {"one", "two", "three", "four", "five"},
307 {
308 __ipairs = function(self)
309 return ipairsaux, self, 0
310 end
311 }
312 )
313 end
314 print(string.concat(",", custom))
315 -- prints: one,two,three
316 -- (note that "four" and "five" are not printed)
318 tbl = {"alpha", "beta"}
320 proxy1 = setmetatable({}, {__index = tbl})
321 for i, v in ipairs(proxy1) do print(i, v) end
322 -- prints:
323 -- 1 alpha
324 -- 2 beta
326 proxy2 = setmetatable({}, {
327 __ipairs = function(self)
328 return "index", proxy1
329 end
330 })
331 for i, v in ipairs(proxy2) do print(i, v) end
332 -- prints:
333 -- 1 alpha
334 -- 2 beta
335 print(proxy2[1])
336 -- prints: nil
338 cursor = setmetatable({
339 "alice", "bob", "charlie", pos=1
340 }, {
341 __call = function(self)
342 local value = self[self.pos]
343 if value == nil then
344 self.pos = 1
345 else
346 self.pos = self.pos + 1
347 end
348 return value
349 end,
350 __ipairs = function(self)
351 return "call", self
352 end
353 })
354 for i, v in ipairs(cursor) do print(i, v) end
355 -- prints:
356 -- 1 alice
357 -- 2 bob
358 -- 3 charlie
359 print(cursor())
360 -- prints: alice
361 for i, v in ipairs(cursor) do print(i, v) end
362 -- prints:
363 -- 1 bob
364 -- 2 charlie
365 -- (note that "alice" has been returned earlier)
367 coefficients = setmetatable({1.25, 3.14, 17.5}, {
368 __index = function(self) return 1 end,
369 __ipairs = function(self) return "raw", self end
370 })
371 for i, v in ipairs(coefficients) do print(i, v) end
372 -- prints:
373 -- 1 1.25
374 -- 2 3.14
375 -- 3 17.5
376 -- (note that iteration terminates even if coefficients[4] == 1)
377 print(coefficients[4])
378 -- prints: 1