Developer Voices on Property-Based Testing Made Me Realize PBT’s Value

I listened to “Automate Your Way to Better Code: Advanced Property Testing (with Oskar Wickström)” and I believe that Property-Based Testing (PBT) clicked for me a bit now. Maybe. It never made sense to me before, but Oskar Wickström has a lot of actually interesting examples.

My highlights are instances where you have to think outside the box, like a good puzzle:

  • You cannot describe complex actions on random data directly (like cutting gaps from audio tracks); but you can offer inverse actions, like undo operations, and then essentially test that applying an action, undoing it, redoing it, then undoing it again doesn’t change the result.
  • Testing a search function, you can’t actually test the search (without implementing the search functionality inside the tests again) because you don’t know the test data. But you can test that filtering the search results works, because filtering should product subsets of the unfiltered search.
  • You can describe web apps as state machines with valid transitions and then go ballistics with trying all kinds of interactions. That’s the promise of Quickstrom: You don’t need to specify each state of the web page as an example, but you specify the properties of each valid state and then ensure that no combination of interactions and button presses produces an invalid state. (Relying on app introspection, this probably works much better on the web with headless browsers than in Xcode/on mobile.)

None of this sounds like a replacement for what unit tests would do. But they do sound like a much better high-level approach to regression testing!

If nothing else, I’m intrigued to learn more about this now.

Can absolutely recommend this episode.

Actually, I recommend the whole podacst.

Async XCTest Assertion Helpers

SwiftAsyncAssert by Angu (@angu@techhub.social):

Instead of writing

import XCTest

func test_should_succeed() async {
    do {
        let isTrue = try await shouldSucceed()
        XCTAssertTrue(isTrue)
    } catch {
        XCFail("Should not throw an error")
    }
}

conveniently write

import SwiftAsyncAssert

func test_should_succeed() async {
    await AsyncAssertTrue(try await shouldSucceed())
}

func test_should_throwError() async {
    await AsyncAssertThrowsError(try await shouldFail())
}

Less code for your convenience when testing!

I ran into this, too, recently and wondered how everyone’s dealing with this. Because this is so annoying, I expected more outcries on the interwebs every day.

Maybe nobody is writing tests?

Either way: These helpers look very nice, and I’ll try them for sure. Should be part of the XCTest libray in my opinion. I seriously wonder why there’s no async XCTest assertion functions for this already.

If you’re just running into this for main actor isolation: One workaround that got suggested to me is to annotate the XCTestCase subclass itself with @MainActor to circumvent having to await isolated calls everywhere. (That doesn’t help for non-main actor-isolated calls, obviously.)

XCTestExpectation via NSPredicate Is On a Slow Interval, Use This Instead

You and I are probably pretty familiar with the XCTestExpectations helper expectation(description:) that we can .fulfill() in an async callback, and then wait in the test case for that to happen. When you have neither async callbacks or notifications handy from which you can create test expectations, then there’s a NSPredicate-based version, too. Like all NSPredicates, it works with key–value-coding, or with block-based checks. These run periodically to check if the predicate resolves to “true”.

Continue reading …

X-Oriented Programming, Using Functions or Classes?

Working on a greenfield project is nice, but it also brings up all the uncertainties of the start: which components, modules, objects, interfaces do you need? With Swift, you even have to decide if you want to design the system in an object-oriented or a functional way. As far as the core functionality is concerned, you do have this choice. (When it comes to UIKit/AppKit, you don’t.)

Continue reading …

Finding the Field Editor in UI Tests

On macOS, the field editor is a NSText instance that appears whenever you edit a NSTextField. This means the text fields themselves offer no editing features; they just tell the shared field editor to appear in their drawing area and show their content. When you write XCUITests, you may want to edit cells in a table or fill out a form with many text fields. Today I learned that you don’t get to the field editor in UI tests and send it the typeText message. You work with the text fields like the user does: as if they themselves accepted user input.

Continue reading …

Being Afraid to Change a Working App

Today I work on The Archive. The focus is on an issue brought up by the community. Search results don’t update the sort order when you modify a note unless you refresh manually. In fact, the issue is expected behavior. The Archive, being a note-taking app where you can filter notes on disk with a live search, is designed to not update the search results for an active search term. Att all. This should prevent the note from disappearing from the results if you remove the search term from its contents. If you search for “foo” and get 10 results, the note you currently edit should not disappear when you cut the search term, “foo”, from it. The Archive protects the search results; a mere live-reload would change the list to 9 results, removing the currently edited one, and that’d be pretty confusing.

Continue reading …

Tests Are Just Code, Too

From “Testing, for people who hate testing”:

The thing is, tests are just code. If you have a hard time constructing your own objects with some particular state, it might be a sign that your API is hard to use!

This is all there is to the magic of test-driven development, we could say. A test is client code. This means it doesn’t have the inside perspective of the object or module under test. It has an outside perspective. Through tests you see how your production code is used (by other parts of your code or by other people in the case of libraries, say).

You start with a test. This means: you start with an outside perspective, asking yourself questions like “What would be a good (public) interface for this?” – The opposite is ad hoc reasoning, something along the lines of “I have this NetworkManager there and maybe just call it and return the JSON. Okay. Oh, now JSON doesn’t work, I need to send a custom object along, so I’ll just parse the JSON and create an object. But if that fails, hmm, I cannot send nil, so I’ll force unwrap and let someone else deal with this.” Or something.
These improvised solutions can cause trouble because all they do is focus on the perceived requirements of the object you’re writing, ignoring the requirements of code that uses the resulting code.

Writing tests first kind of equals writing your app outside-in, starting with the calling code, then implementing the code that’s to be called. Only test cases are super focused and you verify a lot more behavior and setup code with a single run than you could ever do with manual testing.

Experienced programmers will be able to switch perspectives even without writing tests. Their experience helps them to make it less likely to produce waste, whereas “waste” is code you write and then discard because you find it doesn’t fit.

How to Get Started with Swift: Write New Tests in Swift

Daniel Jalkut of MarsEdit fame is slowly getting up to speed with Swift. I have a few Swift-only projects already (Move!, TableFlip and secret in-progress ones). On top of that, every feature I add to the Word Counter is a Swift module, too. The main code of the Word Counter is still in Objective-C, though.

Integrating new types to the project using Swift is awkward. There’re things you can only use in one direction. Most if not all Objective-C code can be made available for Swift and works great. The other way around, not so much. Also, Objective-C imports from the -Swift.h file won’t be available in Swift-based test targets. So you’ll have to isolate these even further, create wrappers, or whatever. Porting a class to Swift can work, but the side effects on testing make it harder.

I like Daniel’s advice to write your unit tests in Swift if you’re still not certain if you should make the switch to this new and volative language. It’s a good idea to get your feet wet.

Considering the problems I ran into with the Word Counter, though, I wouldn’t know how, honestly.

How to Create Fixture Files for Unit Tests

This is mostly a reminder for my future self: It doesn’t suffice to create a TXT file and add it to the target in the file inspector (⌘⌥1) to be able to read it as part of the bundle. You also have to drag it into the “Copy Bundle Resource” build phase of the target so it get, well, bundled into the product. (In my case, it was the test target.)

screenshot of Xcode
The .md file was part of the test target but loading it from the NSBundle failed until I added it to the proper Build Phase
  1. Create an empty file or add an existing file to the target’s group in Xcode and make it part of the proper target,
  2. drag it into “Copy Bundle Resources” build phase,
  3. load it using NSBundle(forClass: TheTestCase.self).URLForResource("filename", withExtension: "txt").

Storing PLIST or JSON files for structured data works just as well. I happen to require plain text flavors most of the time and always wonder why the loading fails for a couple of minutes.

How do You Really Mock Objects You Don't Own? You Replace Them with Adapters

Teaser image

How do you test NSURLSession properly? After all, using it in tests directly will hit the servers; that’s not what you want to do 1000s of times per day. Even if you use a localhost based server replacement the tests will be slower and potentially error-prone as the server replacement may not be 100% accurate. Anyway – there’s a popular series of posts about that topic. There, you’ll learn how to mock NSURLSession in your tests. I think the advice is good to move on quickly, but it only shifts the problem slightly out of sight. There are better alternatives.

Continue reading …

Extract Private Functions as Collaborators

“Using a private function means having a hardwired link to an anonymous collaborator. Over time, this will slowly hurt more.” (@jbrains) I was thinking about this the other day when I wrote tests for a presenter. It receives a date, formats it, and makes the view update a label with the result. Simple. (The real thing actually does a bit more, but that doesn’t matter much.)

Continue reading …

1 Criterion to Determine if You Should Write Unit or UI Automation Tests

Teaser image

Xcode now offers two kinds of tests natively. You don’t have to rely on 3rd party JavaScript libraries anymore to automate the iOS simulator and assert view conditions. So with the advent of native UI Automation tests, can you rely on them to verify your app works? Here’s the two sides of the 1 criterion you’ll ever need to find out:

Continue reading …

Jiggle GCD Queues to Find Problems

To debug my threading issues and help bring forth future problems, I have created a simmple object that slows the current queue down:

let IsRunningTests = NSClassFromString("XCTestCase") != nil

struct QueueJigglePoint {

    /// Randomly interfere with the thread.
    static func jiggle() {
    
        guard !IsRunningTests else { return }
    
        #if DEBUG
            usleep(2*1000000) // 2 seconds
        #endif
    }
}

I got this idea from Brett Schuchert on pages 188–90 of Uncle Bob’s Clean Code. A Handbook of Agile Software Craftsmanship. There, interference with traditional threading should randomly sleep, yield, or fall through. Enqueued blocks are a lot less volatile, so I only came up with sleeping.

Randomizing the sleep interval is up next. But a fixed number of 2–10 seconds helps find UI-blocking code already.

Just throw in a QueueJigglePoint.jiggle() in NSManagedObjectContext.performBlock executions, when dispatching async to the background, or when reading files, for example.

Swift Protocol Extensions as Mixins – And How Do You Test That?

I found a very useful distinction by Matthijs Hollemans to grasp what protocol extensions in Swift 2 can be: instead of interfaces, they are traits or even mixins. At the same time, I want to raise awareness for misuse: just because behavior is mixed-in doesn’t mean it’s additional behavior. The code may live in another file, but its functionality can still clutter your objects.

Continue reading …

How to Write Pragmatic, Testable Singletons in Swift

Singletons have their use. I use two Singletons regularly in my projects: a DomainPublisher and a ServiceLocator. The latter is a registry of service objects which is global so the service objects don’t have to be. For practical use, most Singletons are overcomplicated or overly restrictive. Here’s how I suggest you implement Singletons in your apps:

Continue reading …

Refactoring Legacy Code: Replace Free Function in Tests, Even Swift's Own Functions

Teaser image

Refactoring Legacy Code is hard. There are a few safe refactorings you can do with caution. But most chirurgical cuts require you to put the code in a test harness first to guard against regression. With C and Swift, you can create free functions as part of your app. To verify your objects use that function, you need to find a way to insert a test double.

Continue reading …

Test doubles for Core Data managed objects might not work as expected

You don’t have to learn anything new if you work with Core Data in Swift. It’s a pain to use NSManagedObject directly, so you better work with custom subclasses all the time. Unboxing optional NSNumbers is just too cumbersome. There are caveats you should be aware of, though: Swift’s static typing makes it a bit harder to write test doubles for managed objects.

Continue reading …

Save Your Butt in Unit Tests With Shallow Reaching

I assume you do write tests. To test system boundaries, you have to find out whether methods are called. Usually, people reach out for mocks to verify behavior.

If you use a lot of mocks, though, the tests can reveal high coupling in your code: either you have to mock objects multiple levels deep, which is a sign of too much knowledge about how the collaborating objects work, or you have to mock lots of objects one level deep. The latter can be acceptable when you test a components which is some kind of coordinator. The former, though, is never to be accepted. Use the feedback of these hard-to-write tests to make your code simpler.

That’s what the Law of Demeter is for. While “law” is way too strict, its principle is sound: strive to call methods on the objects you have direct access to, but avoid calling methods on objects which you obtain through another method call in the first place.

For example, the following violates the principle:

func violateThePrinciple(aCollaborator: Collaborator) {
    let aStranger = aCollaborator.obtainFriend() // this is okay so far
    aStranger.doSomething() // this is not, because it's 2nd level
}

You can get by through providing facade methods:

func respectThePrinciple(aCollaborator: Collaborator) {
    aCollaborator.doSomething()
}

extension Collaborator {
    func doSomething() {
        self.friend.doSomething()
    }
}

In your unit tests, you can add a Collaborator mock to verify the calls and then you’re good to go. No multiple levels of mock objects and stubs. You may call this “shallow reaching” (opposed to “deep reaching”), although no one else calls it that.

That’s the Law of Demeter in a nutshell: only talk to friends which you know through properties or parameters, but don’t talk to strangers which you get to know through the return values of your friends’ methods. You never know what to expect.