Friday, June 13, 2014

First experience continued: itertools

In my previous post, I said, "I think you'd have to write the equivalent of the itertools functions (count and takewhile) yourself. As far as I can tell from a quick check, Swift has nothing like Python's yield-based generator functions, or generator expressions; the only way to do it is to create explicit classes that implement the Sequence and Generator Protocols."

Here is where I try to do that.

Sequence and Generator protocols

Swift has the same two basic protocols as Python, just with different names.

In Python, an Iterable is an object with an __iter__ method that returns an Iterator, which is an object with a __next__ method that generates successive values until it runs out, after which it raises StopIteration. An Iterator is itself an Iterable, which returns self for __iter__.

In Swift, a Sequence is an object with a generate method that returns a Generator, which is an object with a next method that generates successive optional values until it runs out, after which it returns nil. A Generator may be a Sequence, in which case it may return self for generate—which seems to be common in the stdlib—but this isn't required.

Creating iterables

There are four different ways to create an iterable in Python:
  1. Write a class with an __iter__ method that returns an instance of a class (possibly self) with a __next__ method. (Or do the equivalent from the C/Java/RPython/etc. layer.)
  2. Write and call a generator function. A generator function is a function with one or more yield (or yield from, but I won't explain that here) expressions. Calling a generator function returns a generator, which is a special kind of iterator. Each time the generator is asked for a value, the function runs until the next yield expression, then generates the value yielded.
  3. Use a generator expression, a type of comprehension that evaluates to a generator.
  4. Create an indexable sequence. (This is primarily a fallback, meant for old code.)
Swift only has the first of these. (Actually, it also has the fourth, but only for creating a Generator that isn't Sequence, as far as I can tell, which isn't all that useful.) This makes things very tedious.

Of course you can call someone else's code that does any of the above. Python's collection types are iterable (and, in some cases, have additional dependent iterables, like dict.items()). It has a couple of builtin functions, map and filter, that take one iterable and give you a different dependent iterator, and its stdlib has many other such functions (mostly in itertools). It also has a wide range of functions that create new iterables (from range to xml.etree.ElementTree.iterparse). And there are third-party libraries to give you even more options. Swift's two collection types give you iterables, and it has map, filter, and an equivalent to range… and that's about it. A large part of the work in Python 3.0 was pushing iteration throughout the language; it's probably the #2 reason (after Unicode) that Python 3 isn't just an incremental change to Python 2. A new language shouldn't have a problem doing this right from the start—as many new languages, like C# and Clojure, have proven. But for some reason, Swift has the kind of minimal iteration support that Python 2.3 had: the framework is there, but most of the functionality you'd want isn't.

But at least the framework is there, so someone could write his own implementation of everything in itertools, etc. And publish it as open source, so everyone else can use it as a CocoaPod or whatever, and then the problem is solved, right? So, let's see how much work it is. (It's worth nothing that some of Python's core developers thing itertools is not particularly important, because most of it can be written trivially with generator expressions or generator functions…)

Count

One of the simplest functions in Python's library is count. The docs show a trivial implementation in Python using a generator function, but even if you had to write a class, it would look as simple as this:
    class count:
        def __init__(self, start=0, step=1):
            self.start = start
            self.step = step
        def __iter__(self):
            return self
        def __next__(self):
            val = self.start
            self.start += self.step
            return val
In Swift, this is complicated by a few things.

The type inference works somewhat backward from the way you'd expect from C++; instead of telling it that you're creating a Generator of Int values, you tell it that you're creating a Generator, make it generate Int values, and it figures out how to fill in the associated types for you. For simple cases like this, that's actually kind of nice, but when you get to more complicated cases, as we'll see below, it's less so.

Also, Swift seems to discourage using classes as if they were functions, so instead of a count class, we'll have to build a Count class and a count function that constructs a Count. (In C++, you would do this for a reason—to allow function template specialization to help you with type inference. In Swift, it seems to be just boilerplate for its own sake.)

Meanwhile, Swift has some nifty features for reducing boilerplate—automatic constructors, optional parameters with default values instead of multiple overloads, etc. But unfortunately, you can only effectively use one of these at a time, because they don't work together nicely.

Finally, in Swift, struct and class have many differences, and it's not always clear which one you want. From the descriptions, a class really sounds like a better idea here—when you pass a Count around, you want the same counter, with shared state, like in Python, right? However, I can't see any way to get any of the more complicated Generator types to work as classes; I either get baffling compiler errors that don't seem to apply to the code at all, or compiler crashes. And most of the stdlib and example types seem to be structs, so… let's go with that. So:
    struct Count: Sequence, Generator {
        var start: Int
        var step: Int
        init(start: Int = 0, step: Int = 1) {
            self.start = start
            self.step = step
        }
        func generate() -> Count {
            return self
        }
        mutating func next() -> Int? {
            let val = start
            start += step
            return val
        }
    }

    func count(start: Int = 0, step: Int = 1) -> Count { 
        return Count(start: start, step: step)
    }

Takewhile

One of the simplest iterator-transforming functions is takewhile. This just generates elements from a source iterable until a predicate returns false. Here it is in Python, as a class (and switching the parameter order, to match what appears to be the opposite standard in Swift):
    class takewhile:
        def __init__(self, iterable, predicate):
            self.iterator = iter(iterable)
            self.predicate = predicate
        def __next__(self):
            val = next(self.iterator)
            if self.predicate(val):
                return val
            raise StopIteration
This one is a lot harder to write in Swift.

First we have to decide whether this takes a Sequence (Iterable) or a Generator (Iterable). Because not every Generator is a Sequence, this could be a potential problem. We could try to take both, but with static typing and no overloads, that's easier said than done. It looks like map and enumerate take a Sequence and return a Generator that's also a Sequence, so let's do that.

Part of this is because of Swift's strange system based on associated types for its protocols—there's no protocol schema Generator, there's instead a single protocol named Generator, which has a typealias named Element, so you need to parameterize on both Generator and T, with a where clause to link them up. This should be somewhat familiar to anyone who's done C++ templates, but very puzzling to anyone who's worked with C#, Java, etc.

The other problem is that, as mentioned earlier, Swift's type inference goes in the easy direction, forcing you to fill in the hard direction. We want to take a Sequence, and act as a Sequence of the same element type. How do we do that? First, we have to parameterize on both the Sequence type and the element type, with a where clause to link them up. Then, our generate function can't figure out its type just because we return self; we have to specify it. Fortunately, in the simple case we're dealing with here, our generator type is just S.GeneratorType; in any more complex case you have to parameterize on a third type that never changes just so you can where-match its associated types (which also means you end up matching on and then exposing a type that your users don't—and, in fact, can't—care about).

Also, if you get anything wrong here, the error messages are baffling, like "type 'TakeWhile<$T0, $T1>' does not conform to protocol 'T'" if you accidentally use Generator where you should have used S.GeneratorType. But at least you don't get 30 lines of template expansion in the error message like most C++ compilers (or, worse, get code that compiles until you try to instantiate it with one of the rare types whose GeneratorType isn't just Generator).
    struct TakeWhile
            : Sequence, Generator {
        var gen: S.GeneratorType
        let pred: T->Bool
        init(sequence: S, predicate: T->Bool) {
            gen = sequence.generate()
            pred = predicate
        }
        func generate() -> TakeWhile {
            return self
        }
        mutating func next() -> T? {
            if let val: T = gen.next() {
                if pred(val) { return val }
            }
            return nil
        }
    }

    func takewhile
                  (sequence: S, predicate: T->Bool) 
                  -> TakeWhile {
        return TakeWhile(sequence: sequence, predicate: predicate)
    }
I have no idea what the idiomatic way is to indent something like this. It's as verbose as Objective C and C++ put together, but all of the examples are so simple that they provide no guidance. But anyway, I think this is reasonably readable.

So, now we can use it. I'll ignore the argv mess, and just compare the Python (not idiomatic Python with generator expressions, of course, but the filter function) and Swift versions of the key part of the code:
    primes = filter(make_isprime(), count(2))
    limited = takewhile(lambda x: x <= limit, primes)
    print(' '.join(map(str, limited)))

    let primes = filter(count(start: 2), make_isprime())
    let limited = takewhile(primes, { $0 <= limit })
    let s = " ".join(map(limited, { $0.description }))
    println("\(s)")

Cleanup

If you exhaust a TakeWhile, it still has a reference to the generator it depends on. If you chain together a whole mess of iterator transformations, they're all going to stay alive as long as the last one does, even after they're consumed. Experience with Python shows that this can be a lot more of a problem with refcounting than with "real" GC, especially once you start using tools like chain.

Final thoughts

From here, it shouldn't be too hard to port the other iterator-transforming tools in itertools. Whether the combinatorial and other functions in itertools, the rest of the stdlib, and libraries like more-itertools will be as easy is another question.

First experience with Swift: primes

When Apple first announced Swift, I downloaded Xcode 6 beta, went through the tutorial/manual, and built a trivial Cocoa app…

But really, building GUI bindings tells you very little about how easy and fun it is to use a language (unless it's just really, really bad); the more interesting thing is to try to code some algorithms and see whether you can think the way you would in a higher-level language, or whether you have to think in more C-like terms.

So, let's generate some primes.

The code for these examples is all at https://github.com/abarnert/swiftprimes.

Sidebar: emacs

A quick google for "emacs swift mode" turned up https://github.com/iamleeg/swift-mode, which requires emacs 24. Apple's built-in emacs is 22.1.1, and Aquamacs 2.4 is 23.4.1, but the Aquamacs 3 alpha is 24.1.50.3, and so far it seems to be working great. So, I'd recommend downloading Aquamacs 3.0a.

Building a command-line app from the command line

First, of course, you have to get swift ready to go. Install Xcode 6.0b1 or later, xcode-select it, then you can use "xcrun swift", both to start an interactive session or to compile a program. You may want to create an alias or shell script so that you can just use "swift" instead.

One interesting thing about the swift compiler is that the default name of a program is the (suffix-stripped) name of the first source file, rather than a.out. So, the Makefile to build a bunch of single-file swift programs can look like this:
    all: foo bar baz

    %: %.swift
    xcrun swift $<
But what about the code? It was a bit hard to find documentation on this. Google told me that C_ARGC is an Int and C_ARGV an UnsafePointer<Cstring> that hold exactly what you'd expect.

An UnsafePointer is how a C pointer (which can, of course, be a pointer to an array) is represented in Swift. So, maybe C_ARGV[1] is a CString? optional string? Nope, it's a CString. So you need to check C_ARGC before accessing C_ARGV. OK, that's fair... but can you put a normal bool expression and an optional-value binding into a single if statement, or do I need a mess of nested if statements here? It seems like the latter.

Well, the mess shouldn't be too bad. String.fromCString(C_ARGV[1]) gives me a String, and String.toInt returns an Int?, so I just need if let limit = String.fromCString(C_ARGV[1]).toInt(), right? No, that gives me an error--first "partial application of a struct method is not allowed" on the ".toInt", and then "bound value in a conditional binding must be of Optional type". Why? I don't know. Splitting everything into three lines works, but at this point, it seems worth wrapping it all up as a function... which is sad, but I'm sure I'll find a better way later. Anyway:
    func intifyarg1() -> Int? {
        if C_ARGC == 2 {
            let limitstr = String.fromCString(C_ARGV[1])
            if let limit = limitstr.toInt() {
         return limit
            }
        }
        return nil
    }

    if let limit = initifyarg1() {
        …
    } else {
        println("\(C_ARGV[0]) LIMIT")
    }

By comparison, the Python equivalent is:
    try:
        limit = int(sys.argv[1])
    except:
        print("{} LIMIT".format(sys.argv[0]))
    else:
        …
But again, hopefully there's a better way to do this.

A simple sieve

One way to generate primes up to some limit is to create an array of LIMIT bools (initially all set to true), then iterate through them, updating all multiples of each from true to false. The easiest implementation relies on being able to mutate the values (but not the shape) of the array as you iterate it. Here's a simple Python implementation, borrowed from http://stackoverflow.com/a/3941967:
    def primes(limit):
        prime = [True] * (limit+1)
        prime[0:2] = False, False
        primes = []
        for (i, isprime) in enumerate(prime):
            if isprime:
                primes.append(i)
                for n in range(i*i, limit+1, i):
                    prime[n] = False
        return primes
So, how hard is this same algorithm to implement in Swift? Pretty easy. I couldn't find any trivial way to create a range with non-unity step so I had to use a C-style for loop, but otherwise, I got it right on my first try, and it looks pretty readable to me:
    func primes(limit: Int) -> Int[] {
        var prime = Array(count: limit+1, repeatedValue: true)
        prime[0...1] = [false, false]
        var primes = Int[]()
        for (i, isprime) in enumerate(prime) {
            if isprime {
                primes.append(i)
                for var n=i*i; n<=limit; n+=i {
                    prime[n] = false
                }
            }
        }
        return primes
    }

Printing the output

In Python, given a list of primes, the obvious way to print it out is:
    print(*primes, sep=' ')
But I wouldn't expect another language to have anything quite that nice. Even Python 2 doesn't have anything that nice. Instead, I'd expect to convert the primes to strings and join them with spaces, then print the result. Like this:
    print(' '.join(map(str, primes)))
There are a few problems trying to do the equivalent in Swift.

First, as cool as string interpolation is, it occasionally gets in the way where a simple duck-typed print function and/or str.format method doesn't. In particular, you can't throw a string literal in the middle of something you want to interpolate. But that's easy to fix; just let space = " " and use that.

Next, there is no str function. The equivalent is a description property on (I think) AnyObject. But I don't know how you map a property (or even an unbound method); just a function. If we wanted to do this generally, we'd want a generic function to convert a Sequence<AnyObject> to a String by joining the description of each object, but that doesn't seem to be as easy as it should be. (This is the first place I've found where C++ has better type inference than Swift...) So let's forget about being generic and just cram an inline function into the map call (using the somewhat-Ruby-blocks-like syntax that allows you to define a function inline as the last argument to another function, without putting it inside the parentheses):
    let result = " ".join(map(primes(limit)) { 
            (var number) -> String in return number.description
        })
I'm not sure of the idiomatic way to indent that (and neither is swift-mode...), but it's not too terrible. Also, I'm not sure having the inline function outside the parens is very helpful when it's inside another function call anyway. Either way, it looks like this may lead to a lot of JavaScript-style }))})})) endings…

Fortunately, Swift has a concise way of writing inline functions that makes things like this a lot simpler, although it's more than a little ugly and perlish:
    let result = " ".join(map(primes(limit)) { $0.description })
Finally, even with the space constant, the string interpolation still fails. And I have no idea why. Apparently the expression is too complicated to fit into an interpolated expression in some way that the compiler can't diagnose for me. Hopefully this is just 1.0 teething troubles.

Generators

What if we wanted to generate a potentially-infinite sequence of primes? We could do that by filtering a sequence of integers with an isprime function. The function needs to store (memoization) state between calls. Python has a few different ways to cram that into the function itself for generality, let's use a closure, because any decent language should be able to handle the equivalent. So:
    def make_isprime():
        memo = []
        def isprime(n):
            for prime in memo:
                if not n % prime: return False
            memo.append(n)
            return True
        return isprime
This is just as natural in Swift as in Python; no stumbling blocks at all writing this on my first try:
    func make_isprime() -> (Int -> Bool) {
        var memo = Int[]()
        func isprime(n: Int) -> Bool {
            for prime in memo {
                if n % prime == 0 { return false }
            }
            memo.append(n)
            return true
        }
        return isprime
    }

Finite generators

Now, how do we use this to find all primes up to LIMIT? Well, we could just filter the closed range from 2 to LIMIT:
    primes = filter(make_isprime(), range(2, limit+1))
What does this look like in Swift? Easy!
    let primes = filter(2...limit, make_isprime())

Infinite generators

But what if we instead wanted to filter an infinite sequence, then truncate it, like this:
    primes = filter(make_isprime(), count(2))
    limited_primes = takewhile(lambda prime: prime <= limit, primes)

(Why would you want to do that? Well, let's say you wanted the first 100 primes, rather than the primes up to 100. You have no way of knowing in advance what input sequence to pass to filter.)

I have no idea how to do this. I think you'd have to write the equivalent of the itertools functions (count and takewhile) yourself. As far as I can tell from a quick check, Swift has nothing like Python's yield-based generator functions, or generator expressions; the only way to do it is to create explicit classes that implement the Sequence and Generator Protocols. Honestly, if that were the only way to create iterators in Python, I don't think the idea would have taken off. That being said, it may be that the answer is a lot more (itertools-like) higher-order functions and a lot fewer explicit generators. Until I actually try it out, it's hard to know how usable it will be. All I can say is that it's nowhere near as simple to do this right out of the box as it is in Python, Ruby, or Haskell.

Printing, again

I tried to print this out using the same code as last time, but whether I put the filter inline, or use a variable, it doesn't seem to work. If you iterate over the filter result, you get all the right value, but if you pass it to map, you get nothing! I have no idea why. So I fell back to using an explicit loop. Ick.
    for prime in filter(2...limit, make_isprime()) {
        print("\(prime) ")
    }
    println("")

Thoughts

Some of the stuff I ran into here is probably based on my lack of knowledge, and some of it probably based on this being the first version of the language. For a compiled, statically-typed language, Swift seems pretty good, but it's not quite as magically Python-/Ruby-like as it at first appears.

Also, I didn't look at all into how powerful the type system is (except for my brief foray into trying to write up a generic joinup function). A statically-typed language with a sufficiently-powerful type system (like Haskell) is every bit as usable as a dynamically-typed language, but with a weak type system (like Java), you end up with horrible casts and checks all over the place, giving you the worst of everything—inflexibility, verbosity, and lack of safety. So, that's a pretty important question, but I'll get to it later.