iterrr!

iterate faster ... 🏎️. Write higher-order functions, get its imperative style at the compile time!

The Problem

The problem is that writing a full nested loop is a boring task, and using clojure iterators slows down the speed.

(std/sequtils is a nightmare, iterutils is slightly better, but can we go faster? )

The real question is: "Can meta-programming help us?"

The Solution

iterrr uses the ultimate power of meta-programming to bring you what you've just wished.

example of generated code

"hello".pairs |> 
  filter((i, _) => i  > 1)
  .map((_, ch) => ch)
  .strjoin() ## llo

block:
  template iterrrFn3(_; ch): untyped {.dirty.} =
    ch

  template iterrrFn4(i; _): untyped {.dirty.} =
    i > 1

  var iterrrAcc2 = strjoinInit[typeof(iterrrFn3(
      default(typeof("hello".pairs))[0], default(typeof("hello".pairs))[1]))]()
  block mainLoop:
    for li1 in "hello".pairs:
      if iterrrFn4(li1[0], li1[1]):
        block:
          let li1 = iterrrFn3(li1[0], li1[1])
          if not strjoinUpdate(iterrrAcc2, li1):
            break mainLoop
  strjoinFinalizer(iterrrAcc2)

Usage

complete syntax

There is 3 type of usage:

# predefined reducer
iterable |> entity1(_).entity2(_)...Reducer()

# custom reducer
iterable |> entity1(_).entity2(_)...reduce(loopIdents, accIdent = initial_value, [Finalizer]):
  # update accIdent

# custom code
iterable |> entity1(_).entity2(_)...each(...loopIdents):
  # do with loopIdents

Main Entities:

1. map :: similar to mapIt from std/sequtils
2. filter :: similar to filterIt from std/sequtils
3. breakif :: similar to takeWhile in functional programming languages but negative.
4. inject :: injects custom code

1. predefined reducer

NOTE: you can chain as many map/filter/... as you want in any order, but there is only one reducer.

There are some predefined reducers in iterrr library:

• toSeq :: stores elements into a seq
• count :: counts elements
• sum :: calculates summation
• min :: calculates minimum
• max :: calculates maximum
• first :: returns the first item
• last :: returns the last item
• any :: similar to any from std/sequtils
• all :: similar to all from std/sequtils
• toHashSet :: stores elements into a HashSet
• strJoin :: similar to join from std/strutils
• toCountTable :: similar to toCountTable from std/tables

here's how you can get maximum x, when flatPoints is: [x0, y0, x1, y1, x2, y2, ...]

let xmax = flatPoints.pairs |> filter(it[0] mod 2 == 0).map(it[1]).max()
# or
let xmax = countup(0, flatPoints.high, 2) |> map(flatPoints[it]).max()

NOTE: see more examples in tests/test.nim

Custom Idents ?!?

using just it in mapIt and filterIt is just ... and makes code a little unreadable.

remember these principles when using custom ident:

1. if there was no custom idents, it is assumed
2. if there was only 1 custom ident, the custom ident is replaced with it
3. if there was more than 1 custom idents, it is unpacked

Here's some examples:

(1..10) |> map( _ ) # "it" is available inside the "map"
(1..10) |> map(n => _ )
(1..10) |> map((n) => _ )
(1..10) |> map((a1, a2, ...) => _ )
(1..10) |> reduce((a1, a2, ...), acc = 2)
(1..10) |> each(a1, a2)

example:

"hello".pairs |> filter((i, c) => i > 2).map((_, c) => ord c)

Limitation

you have to specify the iterator for seq and other iterable objects [HSlice is an exception]

example:

let s = [1, 2, 3]
echo s |> map($it).toseq() # doesn't work
echo s.items |> map($it).toseq() # works fine
echo s.pairs |> map($it).toseq() # works fine

Define Your Reducer!

every reducer have: [let't name our custom reducer zzz]

1. zzzInit[T](args...): ... :: initializes the value of accumulator(state) :: must be generic.
2. zzzUpdate(var acc, newValue): bool :: updates the accumulator based on newValue, if returns false, the iteration stops.
3. zzzFinalizer(n): ... :: returns the result of the accumulator.

NOTE: see implementations in src/iterrr/reducers.nim

Custom Reducer

pattern:

ITER |> ...reduce(idents, acc = initial_value, [finalizer]): 
   update acc here 

Notes:

• acc can be any ident like result or answer, ...
• Finalizer:
  • it's optional
  • it's an experssion inside of it you have access to the acc ident
  • the default finalizer is acc ident

Example of searching for a number:

let element = (1..10) |> reduce(it, answer = none int, answer.get):
  if your_condition(it):
    answer = some MyNumber
    break mainLoop

Note: if the item has not found, raises UnpackDefect error as result of get function in finalizer answer.get.

Don't Wanna Use Reducer?

My view is that a lot of the time in Nim when you're doing filter or map you're just going to operate it on afterwards :: @beef331 AKA beef.

I'm agree with beef. it happens a lot. you can do it with each(arg1, arg2,...). [arguments semantic is the same as custom idents]

(1..10) |> filter(it in 3..5).each(num):
  echo num
  if num < 7:
    break mainLoop

Note: mainLoop is the main loop block

Custom Adapter

adapters are inspired from implmentation of iterators in Nim. TODO: explain more

Limitations: you have to import the dependencies of adapters in order to use them.

Built-in adapter:

• group
• window
• cycle
• flatten
• drop
• take

Usage: example:

let matrix = [
  [1, 2, 3],
  [4, 5, 6],
  [7, 8, 9]
]
matrix.items |> flatten().map(-it).cycle(11).group(4).toseq()

result:

@[
  @[-1, -2, -3, -4], 
  @[-5, -6, -7, -8], 
  @[-9, -1, -2] 
]

see tests/test.nim for more.

Define your custom adapter: for now the name of loop iterator are limited to it. TODO; see src/iterrr/adapters.

Debugging

use -d:iterrrDebug flag to see generated code.

Breaking changes

0.x -> 1.x:

• using brackets for defining custom idents is no longer supported.

Inspirations

1. zero_functional
2. itertools
3. slicerator
4. xflywind's comment on this issue
5. mangle

Common Questions:

iterrr VS zero_functional:

iterrr targets the same problem as zero_functional, while being better at extensibility.

Is it "zero cost" like zero_functional though?

well NO, most of it is because of reducer update calls, however the speed difference is soooo tiny and you can't even measure it. I could define all reducer updates as template instead of function but IMO it's better to have call stack when you hit errors ...

Quotes

writing macro is kind of addicting... :: PMunch