XCTestCase#waitFalseExpectationUntilTimeout implementation

I am working on an iOS project and I'm in charge of testing most parts of it. As I write some tests, I often have the need to wait for an asynchronous method to finish, and then test that something did not happen.

For example, try to login with an invalid username and password, wait until the network communications are finished, then test that the user was not able to login

Swift has some nifty methods for testing these sort of things; the ones I use the most are expectationForPredicate and waitForExpectationsWithTimeout. These methods are good for testing that something happened after the asynchronous method finishes. However, to test that something did not happen (which is what I want), there doesn't seem to be anything designed specifically for this.

• I can't use predicates because that will cause the test to timeout early

• I cannot modify the code being tested because it is an app lifecycle callback and we testers generally do not modify any code, we just test

Which is why I designed this method

extension XCTestCase {
    func waitFalseExpectationsUntilTimeout(timeout: NSTimeInterval, expectations: [XCTestExpectation], handler: XCWaitCompletionHandler?) {

        class CompleteSelector {
            @objc func fullFillExpectations(timer: NSTimer) {
                let expectations = timer.userInfo as! [XCTestExpectation]
                for expc in expectations {
                    expc.fulfill()
                }
            }
        }

        let smaller = timeout < 0 ? 0 : timeout.advancedBy(-0.5)

        NSTimer.scheduledTimerWithTimeInterval(smaller, target: CompleteSelector(), selector:
            Selector("fullFillExpectations:"), userInfo: expectations, repeats: false)

        waitForExpectationsWithTimeout(timeout, handler: handler)
    }
}

The main thing the function does is to schedule a timer and fulfill all expectations just before the waitForExpectationsWithTimeout method times out. Is this the best way to deal with th

This is an extremely overcomplicated solution to a relatively straightforward problem.

The waitForExpectationsWithTimeout method should effectively only be used assure that a particular asynchronous expectation was fulfilled.

So, in the case where we are testing whether or not the user was successfully logged in, it'd be tempting to write something along the lines of:

func testLogin() {
    let loginExpectation = expectationWithDescription("User successfully logged in")

    LoginManager.doLogin { success in
        if success {
            loginExpectation.fulfill()
        }
    }

    waitForExpectationsWithTimeout(10, handler: nil)
}

But problematically, this test can fail for two reasons:

• The user wasn't successfully logged in.

• The expectation wasn't fulfilled in the time limit.

If the test fails, we can't be sure whether it would have passed given a longer time limit or if it returned very quickly but wasn't successful.

So, we need to modify our test to look more like this:

func testLogin() {
    let networkExpectation = expectationWithDescription("Network request completed")

    LoginManager.doLogin { success in
        XCTAssertTrue(success)
        networkExpectation.fulfill()
    }

    waitForExpectationsWithTimeout(10, handler: nil)
}

Now, the test determines whether or not the asynchronous code executed, and if so, we verified whether or not the login was a success (or whatever the success variable represents here).

Now, you're looking for the opposite case. You want to know not whether or not the user logged in, but whether or not the user was prevented from logging in. And that's as simple as inverting an XCTAssertTrue to an XCTAssertFalse.

func testLogin() {
    let networkExpectation = expectationWithDescription("Network request completed")

    LoginManager.doLogin { success in
        XCTAssertFalse(success)
        networkExpectation.fulfill()
    }

    waitForExpectationsWithTimeout(10, handler: nil)
}

Importantly, this pattern doesn't require any complicated code (that we would have to of course write tests around ourselves). Our tests are using stuff purely out of the Xcode test suite box--exactly what any one looking at our code base would expect to see. It's very straight forward what is going on here.

And we are able to distinguish between whether the asynchronous call back ever happened or if it did happen but we got a bad result. There's a huge difference here between a test failing because the asynchronous code didn't run before our time out and it did run but it gave an incorrect result. We need to clearly distinguish between these results, and doing so doesn't require writing any complicated code whatsoever.

An important note here, make sure that fulfilling the expectation is the very last thing you do in the asynchronous callbacks, or you may set up a race condition.

And finally, your tests should certainly be far more robust than this. Ultimately, we should be testing with mocked network responses and we should know, for each test, exactly what network response we will get. We should test that whatever code is doing the login returns back the correct information to us. There's a difference between the server allowing the user to log in successfully and denying the user with invalid credentials. There's also a difference between invalid credentials and getting any number of various server errors (most commonly, errors in the 5xx range). Getting a 503 and getting an invalid credential response are far from the same thing, and your tests should validate you're getting what you expect to get.

So, I've basically just told you to completely scrap all the code you've written. And I'll stand by that. There's no reason for the code you've written to exist. But, we can still talk about some of the issues your existing code does have.

This function name has a weird name: waitFalseExpectationsUntilTimeout. It's hard for me to tell by name alone what to expect this method to do. It looks similar to the existing waitForExpectationsWithTimeout(timeout: NSTimeInterval, handler: XCWaitCompletionHandler?) method, but it has this weird array of expectations I'm supposed to pass in. The name needs to be better. Something like assertUnfulfilledExpectationsWithTimeout.

Even with the better name, we're in extraordinary trouble here. As far as I can see, there's no public means of checking whether or not an expectation has been previously fulfilled. Your code doesn't really do anything beyond fulfilling all of the expectations in the last moment before the test completes. It doesn't assure that they weren't fulfilled earlier.

let smaller = timeout < 0 ? 0 : timeout.advancedBy(-0.5)

This variable deserves a better name, and this logic either deserves a comment or it should be eliminated altogether. I'd opt for eliminating it altogether and throwing an exception if the user has passed a value less than 0.

StackExchange Code Review Q#123701, answer score: 3