Realm takes a reactive, Objects-as-APIs approach to dealing with data. In this Realm Meetup talk, JP Simard goes over tips and tricks for building a variety of user interactions by building off a few primitive Realm notification concepts. We’ll cover Interface-Driven Writes, asynchronous versus synchronous query computation, how to architect your app to handle changes that can occur at arbitrary times, and how all of this ties into the Realm Platform.

---

Introduction (00:00)

I want to talk about how to make the most of notifications in Realm.

Previously, we heard Tim talk about how to use objects as messages, as events, or as APIs, or use them as a transport layer to talk to your server. This is something that we’ve been pushing for at Realm, not just between device and server, but even within a device. That’s what I want to talk about today.

Notifications let you respond to changes. Here we have our RealmTasks sample app that shows you the application state synchronizing across different instances of the app; in this case, a Mac app and an iOS app. It has some nice, nifty animations, it’s very fluid, and you have instant feedback when you’re manipulating the objects in the app.

I’ll show how you can use Realm notifications to build rich experiences like these, and we’ll see that Realm has a number of ways that it can notify you of changes. I’ll talk about when you should use one over the other, how to use one over the other, and some new stuff that we have coming up.

As Apple platform developers, we’re all familiar with the different ways to send messages in Cocoa. You have notifications, callback blocks, setters; you can do dependency injection, KVO, protocols. You have all sorts of ways to notify other interested parties that something relevant to them has happened, and they should update their state, or they should reflect whatever happened.

Reactive Programming (02:26)

At Realm, we believe the best way to build an app is to use this pattern of reactive programming. You may have heard the word “reactive” before used in functional reactive programming; and although that certainly has its place, it can be scary to beginners. It can throw you for a loop if you use a more imperative style. If you’ve seen these massive, infinitely-nested stack traces when you’re working with functional reactive programming, you might not necessarily want to dive into it. I want to make clear that reactive programming is not necessarily functional programming.

Reactive programming just means reacting to the changes that are happening, and trying to minimize the number of different code paths they have to maintain to notify interested parties that changes have happened. Minimizing those code paths and aligning everything will simplify your life as a developer and make your code more maintainable.

It’s also about avoiding stale data. We’ve all done this before, where you’re doing some dependency injection, and you pass a value to the next view controller that you want to load up. Then, maybe a fraction of a second later, that data changes. In that case, this concept of dependency injection is not enough. You need to go a little further.

We’ve all been guilty of implementing different code paths to essentially do the same thing. Say you have your UI, and you can update it in a number of ways–a user can add an entry to a list, or that entry can come in by another thread in the app, or through the database, or through the network. We’ve all built these separate code paths in our app to accommodate all these different ways that things can change: networks, database, user action, push notification, system alert, other processes interacting with your app (i.e., extensions in iOS 8 and later). Wouldn’t it be nice if all of this all went through the same code path?

It would save time and make our code simpler to maintain and write. That is essentially Realm’s ideal. We’re pushing for this concept of live objects and auto-updating results, so that you have your one source of truth and everything is coming in through this unidirectional data flow.

You don’t have to adapt to different ways that your data can change. All you care is that it did change, and that you’re notified exactly what about it changed. That way you don’t have to steal data. You don’t have to worry about pushing versus pulling. You’re just told every time something changes. It’s a much simpler conceptual model of things, the arrow of time, always going in the same direction.

Intro to Realm Notifications (05:51)

Let’s look at how Realm exposes different ways you can have notifications. We have this open source sample app with implementations for Mac OS, iOS, Android, and an upcoming version for C# .NET, as well, for Xamarin. This is a hands-on, real-world example of a number of different ways to leverage Realm notifications to build the user experience that you want.

Realm offers a handful of ways that you can be notified of changes:

• Realm Notifications: This is the very first one that we offered, shortly after we launched over two years ago, for Objective-C. This notification will let you know that Realm somehow, somewhere, has changed. It won’t tell you what changed, but will tell you that it has changed.
• Collection Notifications will let you know when a collection of objects has changed.
• Key-value observing.
• Upcoming APIs that will allow some other use cases.

Realm Notifications (07:28)

let realm = try Realm()
let token = realm.addNotificationBlock { notification, realm in
    viewController.updateUI()
}

// Later...
token.stop()

Very simple: you initialize your Realm, and you can add a notification block. Very few lines of code. You end up getting the notification type and the Realm instance in that callback log. From there, you’re not told what changed, in what capacity, or to what extent. It could be that all of your data is gone because you wrote “Realm delete all”, or it could be that you just changed a single property–incremented an integer, for example. You don’t know.

This is useful when you want to react to anything that has happened. There is a handful of reasons why you might want that: if you want to re-serialize the database in another format for whatever reason, you can subscribe to those notifications and be notified every time something changes. That way you don’t have to go through all the potential collections and subscribe to those individually, if all you care about is has anything changed?

If you have a very simple app, it might be easier for you to implement this. You don’t have to think too much about how to query your data. You can update your whole UI, update your whole state, whenever that happens.

Keep in mind that these are delivered asynchronously. Typically, this is done every run loop iteration. If you’re on the main thread, it happens every few milliseconds, and you’ll only be notified once something changes. But there’s usually a better way to subscribe to notifications, because odds are you’re storing more than one type of thing in your Realm, and when you subscribe to notifications, you probably know exactly what change you should be listening for.

The token variable that we are assigning to is a return value from addNotificationBlock. That’s essentially an identifier for that specific notification block. If you have other Realms, or if you call this method several times, it will return different tokens.

Notification Tokens are objects that tell Realm how long it should be delivering these notifications for. For example, you may only have a short window of time where you care about things changing. That way Realm doesn’t go through the internals of preparing notifications, watching for notifications, going through all the notification blocks and then sending them, if you don’t care about it anymore.

It’s very important to call stop() when you no longer want to be notified. This could be on viewWillDisappear if you only care about being notified for that view controller, or this can live for the duration of your app. It depends on how long you care to receive these notifications.

You have to hold a strong reference. The reason we return anything at all is to make sure that a user doesn’t accidentally type in addNotificationBlock without assigning the value to anything; then we’ll just think that the token has de-allocated and we should not be sending these notifications to you anymore. Very important: hold a strong reference to it for as long as you want these notifications.

Collection Notifications (11:30)

let realm = try Realm()
let results = realm.objects(Item.self) // Auto-Updating Results
let token = results.addNotificationBlock(tableView.applyChanges)

Just a single method: addNotificationBlock, and this works with any Realm collection type. It works with Lists, which is a property type, the Swift equivalent to RLMArray. It works with Results, which is the query container in Realm. It works with the type-erased container for all of these containers, any Realm collection. You can use this with linking objects, which is the inverse property type.

All of these different collection types have this addNotificationBlock, and this is where the real magic happens. This is where you get fine-grained change set information.

extension UITableView {
  func applyChanges<T>(changes: RealmCollectionChange<T>) {
    switch changes {
      case .initial: reloadData()
      case .update(let results, let deletions, let insertions, let updates):
        let fromRow = { (row: Int) in return IndexPath(row: row, section: 0) }
        
        beginUpdates()
        insertRows(at: insertions.map(fromRow), with: .automatic)
        reloadRows(at: updates.map(fromRow), with: .automatic)
        deleteRows(at: deletions.map(fromRow), with: .automatic)
        endUpdates()
      case .error(let error): fatalError("\(error)")
    }
  }
}

Say you have a collection, and you want to know which of those objects in the collection–whether it’s Results, a List, or linking objects–were moved, removed, updated, and inserted.

This is where this Realm collection change type comes in. It’s an enum that has three states: initial, update, and error.

The initial state will get triggered as soon as the results are available. The update state will always happen after the initial one. The initial one is always delivered, and always delivered first. The update is sent every subsequent time after that.

The update contains this state change information that you can apply to something else where you are tracking stage, such as a UITableView, or UICollectionView. You can apply this diff and get nice system-provided animations, like for UITableView.

The error case is rare: that should only happen if an iOS device is out of available virtual memory space, and we couldn’t create the new thread where we’re computing these changes.

The crux of this is the initial and the update states; the initial one being the first one, and then the update being all subsequent ones with patch information.

A few gotchas: Notifications can be coalesced. They can be grouped together.

switch changes {
  case .initial: reloadData()
  case .update(let results, let deletions, let insertions, let updates):
    let fromRow = { (row: Int) in return IndexPath(row: row, section: 0) }

    beginUpdates()
    insertRows(at: insertions.map(fromRow), with: .automatic)
    reloadRows(at: updates.map(fromRow), with: .automatic)
    deleteRows(at: deletions.map(fromRow), with: .automatic)
    endUpdates()
  case .error(let error): fatalError("\(error)")
}

Not every write transaction means that you will get a one-to-one notification block invitation. For example, if you have several background threads that are all writing tons of data at the Realm, in many different transactions, it’s quite likely that many of those transactions are bundled together. This is where Realm does the heavy lifting for merging them together so that you have a single change object.

Say you had 1,000 transactions happen in a run loop iteration. You don’t have to apply 1,000 little diffs to your UITableView, which will drastically slow down its performance. You just have to apply one of them; they will be grouped together.

This can be surprising when the initial notification contains more than what the results included when you added that block. You can get your query, your results, call addNotificationBlock and say, at that point, it has zero objects. Right after that, you add an object; it might be that the initial notification gets that one object.

Keep in mind that the initial notification is delivered asynchronously (like all notifications here). It may include more than what you expect, but it will always have the latest state information: it will not be stale. The same thing happens for the update element.

If you’ve used Realm with Java, you may be familiar with the difference between synchronous and asynchronous queries. The way we handle things with Realm Cocoa is that results are queried asynchronously: they’re always processed on a background thread. This includes results that are handled on the main thread. They’re processed on the background thread, so even if you’re waiting for it, you are not blocking the main thread.

Let’s look at what it would look like to never block the source thread when you are computing a query.

let realm = try Realm()
// Query never performed on the current thread
let results = realm.objects(Item.self)
let token = results.addNotificationBlock { _ in
  /* results available asynchronously here */
}

We’re getting all of the item objects in this Realm. Because we’re not accessing any of the elements from the results, we’re not computing. We are not performing that query on the main thread synchronously at that point in time.

This notification block will only be triggered once the background thread has had a chance to perform the query and deliver it back to this thread that we are on, which usually is the main thread. Even if you have a million objects in your Realm, all of the work that goes into performing this query does not block your main thread. That’s extremely useful!

However, if you do need to access an item from the results right away, we’ll take that as a sign that you need this synchronously.

let realm = try Realm()
// Query performed synchronously to return first result
_ = results.first // <- query performed here
let token = results.addNotificationBlock { _ in
  /* results available asynchronously here */
}

If you did not need it synchronously, you wouldn’t be making this synchronous call, and Realm tries to do the smart thing here. In this case, since we’re trying to access one of the elements in the results, that’s where the query will be performed. But even in this case, all of the auto updating of these results across run loop iterations, across time, will happen in the background. This is a nice optimization that we do for you.

To summarize Collection Notifications:

• Can be coalesced, it does not always match one-to-one for the right transactions.
• Generally delivered on the next run loop iteration.
• If you have a background thread that committed a write, odds are that the next time the main thread run loop comes around, it will have that change. Sometimes there’s a bit of a delay, but it’s essentially whenever that change is available for that thread. All of the heavy lifting is done on a background thread to save you from yourself.
• Notifications are always delivered on the source thread, even though it’s delivered asynchronously. That addNotificationBlock runs in the same thread that you called it on.
• Finally, if you’re careful, and if you understand how this works, you can avoid ever blocking a main thread.

Interface-Driven Writes (20:11)

We’ve talked about mostly ways to asynchronously be notified of changes. Although that’s great in most cases, where you don’t want to block the main thread for user interaction reasons, there are times where you do need to do what we call interface-driven writes. That means changes that are done by the user, typically, that need to be applied synchronously in the user interface.

For example, this could be reordering a UITableViewCell, or tapping a button and not wanting to wait those few milliseconds (a few run loop iterations) to see the result. In that case, intuitively, what you want to do is make that change in data on the main tread, and then immediately mirror it as part of that same local code in the UI, without having to wait for the asynchronous notification (which may come at any point in the future).

But the whole point of change set notifications, and this update aspect of Realm collection change, is to give you this patch–this incremental state difference–saying, last time I delivered a notification, we were at point X in the state. Next notification, we are at point Y. Notification after that, point Z, and you want to incrementally go from X to Y to Z.

If you bypass this mechanism and update the state, say in your UITableView, then all of a sudden this incremental diff doesn’t make sense anymore. The notification block will deliver you something like, we noticed that you added an entry to your list, but you’ve already mirrored that in the table view. When we say, let’s add one more, you’re getting that twice, and then UITableView throws an exception: NS Internal Inconsistency State Exception. And then you’re sad, because your app crashes!

We’re in a pickle here: on one hand, we’re saying, we would love to have this unidirectional data flow; on the other, we have to work with tools, such as UITableView, that do not have a unidirectional flow where you can make these synchronous changes. Some of these changes need to be applied to the UI synchronously. Again, reordering rows in a TableView is one of these.

func viewDidLoad() {
  messages = realm.objects(Message.self)
  self.token = messages.addNotificationBlock(tableView.applyChanges)
}

func tableView(_ tableView: UITableView,
               commit editingStyle: UITableViewCellEditingStyle,
               forRowAt indexPath: IndexPath) {
  guard editingStyle == .delete else { return }
  realm.beginWrite()
  messages.removeAtIndex(indexPath.row)
  tableView.deleteRows(at: [indexPath], with: .automatic)
  realm.commitWrite(withoutNotifying: [self.token])
}

Here we want this reordering operation to happen in real time. We think the right solution to this problem is the ability to skip notification blocks.

In this case, I’m adding a notification block to my results of Message, all of the messages in the Realm. This notification block will be delivered asynchronously whenever it’s available, auto-updating.

Say the user wants to delete one of the rows. I want to mirror that instantly on the main thread. I don’t want to wait for the asynchronous call. I start a write transaction, I remove the item from the Realm list, I mirror that in the UI right away (TableView.deleteRows()). I get a nice animation, and then I call commitWrite without notifying this token, and what that means is you are not notifying that notification block.

Otherwise, what would happen is a number of things. Say I was not using the withoutNotifying variant of commitWrite. The UI would start the automation, and shortly after that, the asynchronous notification would be delivered. If you choose to apply the update patch, you will get this internal inconsistent state exception. You don’t want that. Maybe you’re doing some clever workaround, detecting if the state is inconsistent, and reloading the data, but then you’ll lose the nice animation because it’s still in the process of happening. We found ourselves struggling with this as we were building Realm tasks.

That’s where we came up with the write-skipping-notifications concept. This should be constrained to the main thread when you’re responding to user-driven changes that need to be mirrored synchronously. That’s our recommendation. And at that point, we skip that notification. We know that whatever internal state you’re tracking, such as UITableView, is already up-to-date.

Key-Value Observation (25:15)

Key-Value Observation (KVO) is a brand-new API from Apple. We chose to support it because Realm wants to fully buy in to the ecosystem and the tools that people are already familiar with, even if some of them are inelegant or rough around the edges. That was the motivation for adding KVO support.

KVO enables Realm to work with Cocoa bindings on the app kit level, and to work with ReactiveCocoa and other things that track state via KVO. KVO notifications are delivered synchronously, and this is where you get this change in semantics, where as soon as the property of a KVO-observed object is modified, you will get that synchronous notification in your KVO observer method.

However, it does have the disadvantage of only being able to observe a single property at a time. You can loop through the properties of your object and add KVO observers for all of them, but that’s quite a bit of overhead, that ideally would be nice to get rid of.

Objects as Messages (28:48)

We can use this notification functionality to use objects as messages–using objects as ways to drive events. It’s like an event queue. For example, say you have an iOS app extension, and you want to request a response from the main app. Interprocess communication on iOS is tricky, but the good news for you is that Realm has already solved this for multiple processes accessing the same Realm.

You can use Realm objects as messages, or as APIs, where you can just drop, for example, a message or event or request object into a Realm that’s shared between processes. The other process will see it, if it is listening to notifications, and then it can react on that: it can convert an image to text, spin off a network request if it needs to, and then write back to the same local object that it saw from its notification–just write a change to one of the properties (i.e., a response property, or a status property), and then immediately, the other process will see that.

It’s a nice way to build this interprocess messaging system, and it doesn’t only need to be used between processes. This is exactly the use case that enables talking to the Realm object server. It’s what enables massively distributed apps to talk to each other, while still just dealing with local objects, because you’re getting local notifications for remote changes.

Here’s a concrete example of how we use this ourselves in the Realm APIs.

let managementRealm = try user.managementRealm()
try managementRealm.write {
  let permissionChange = SyncPermissionChange(realmURL: realmURL,
                                              userID: anotherUserID,
                                              mayRead: true,
                                              mayWrite: true,
                                              mayManage: false)
  managementRealm.add(permissionChange)
}

We recently added access control to the Realm object server and the Realm Platform. Say you have this Realm that you can sync between all of your other devices, but still just a single user. If you want to share that with someone, you can drop this permissionChange object into the managementRealm.

This Realm is synchronized. It’s unique to you, as a Realm object server user. The object server has notifications, and it will look for these objects that are dropped in this managementRealm and then apply the changes that you pass in, in your fully type-safe model (you have mayRead, mayWrite, and mayManage as access control modifiers).

Now, as a client, as soon as you’ve dropped this object into the Realm, you can then subscribe to it for notifications.

let collectionOfOne = managementRealm.objects(SyncPermissionChange.self)
                                     .filter("id = %@", permissionChange.id)
token = collectionOfOne.addNotificationBlock { notification in
  if case .update(let changes, _, _, _) = notification,
    let change = changes.first {
    // Object Server processed the permission change operation
    switch change.status {
    case .notProcessed: break // handle case
    case .success: break // handle case
    case .error: break // handle case
    }
    print(change.statusMessage) // contains error or message
  }
}

We could use KVO to observe the status property of the permission change object, but the downside to KVO is you’ll get those notifications synchronously delivered. We want these notifications asynchronously delivered, so we’re not blocking the main thread.

We treat this single object as a collection, but it’s a collection of one. Add the notification block, and then once you get that object back, look at the first object’s status. (It’s a collection of one.) There you can see how the server responded to this “event” object and print the status message.

At that point, you’ve created your own callback mechanism by using the same concept that you apply for local objects, for objects that are synced between you and the server and other devices.

This is not exactly elegant, because you have to create this collection of one and this ephemeral result. Wouldn’t it be great if you could just add notification blocks to objects themselves?

The good news is that we’re working on that, and will have something for you in the next few months, early 2017. This is what it will look like.

Object-Level Notifications (31:27)

let realm = try Realm()
let jane = realm.objects(Person.self).filter("name == 'Jane'").first!
let token = jane.addNotificationBlock { change in
  switch change {
  case .change(let propertyChanges):
    for propChange in propertyChanges {
      print("'\(propChange.name)': \(propChange.oldValue) -> \(propChange.newValue)")
    }
  case .deleted: print("object was deleted")
  case .error(let error): print("notification couldn't be delivered: \(error)")
  }
}

// Later...
token.stop()

Say you have this Person object–her name is Jane. All I care is whether or not that specific object has had changes. This is useful if you have something like an appState object, and you want to know when one of its properties has changed. You can update your UI, have an unread notifications badge. You can observe that one property, and have those changes asynchronously delivered. You can also be notified whenever that object is deleted.

This concept of a notification token, and the ability to have write transactions that skip notifications, come in handy if you do need to synchronously update that UI. This is the API that we’re working on, with a handful of other variants.

let realm = try Realm()
let state = realm.objects(AppState.self).first!
let token1 = state.addNotificationBlock(observingProperties: ["unreadCount", "status"],
                                        block: { change in
  // handle change and restrict to properties we care about
})

let token2 = state.addNotificationBlock(ignoringProperties: ["noisyProp"],
                                        block: { change in
  // handle change and avoid being notified for properties we don't care about
})

If you only care about a handful of properties in your object, you can whitelist those and say, only notify me when these things change. You’re not notified for things that you don’t reflect in your UI, for example. The same thing goes if you want to blacklist some property. Say you have a property that’s constantly updated for whatever reason. That’s a noisy property, and you don’t mirror it in the UI; you want notifications for everything but that.

These are the APIs that we’re working on for early 2017. They’ll be asynchronously delivered, like the Results notifications, and should drastically simplify those use cases of objects as APIs.

That, in a nutshell, is a grand tour of Realm’s notification capabilities.

Resources (33:32)

If you want to read more:

• Realm has docs on notifications.
• There’s a great blog post from Marin Todorov on live objects and fine-grained notifications and how to architect your app to make the most of this setup.
• There’s a RealmTasks app, which is entirely open source. It uses the Realm Platform and has hands-on practical, pragmatic examples of how to use notifications. There’s one particular pull request that I encourage you to look at if you’re trying to grok what this interface-driven writes functionality is all about…
• It’s this PR that we were happy to finally land when we had that API available. We were able to get rid of many workarounds that we had in place, and many of these workarounds were complex. They were hard to maintain, they had compromises where if, in the notification block, you realize that the state’s inconsistent, you just call reload data, and then you lose the nice animation. Take a look at that real-world example.
• There are tools that make using Realm and Realm notifications easier if you’re using tools like RxSwift, or ReactiveCocoa.

Q & A (35:05)

Q: Say you had a app like WhatsApp, and you had the check marks for unread and read. Because you are scrolling through a UI collection of your UITableView, is it a good idea, in a self or indexPath, to put a notification block there as you are scrolling around?

JP: One thing that you would have to make sure to do is–if that notification is just for an object in a cell–whenever that cell gets dequeued, make sure you’re doing proper notification lifecycle management. Call stop on the notification token when that cell gets dequeued and you don’t care about being notified on that anymore.

But, taking a step back, generally these messaging apps that have a read status or read receipt only have a single one for a whole conversation. In that case, you might model it as a conversation object that has a single property: “last read message” or “last read date” or something like that. And I would recommend handling that this way: when you’re in that view controller, you have the whole conversation with maybe a List property for all of the messages in that conversation. This conversation object would probably have that “last read date” property. Then you could KVO observe that, or use the object-level notifications with that one property when we have that available.

Q: There are cases where, for example, in iMessage or Facebook Messenger, you send three messages but the first one out of the three failed, or there was an issue submitting it. Would you want to show the user something like that?

JP: Yes. That hits on a great point for the advantages of using a full database locally for things like messages and events. Say you were making a REST call to post this message, or to send this message, and your network connection failed halfway through. It would be tricky for you to know if the message had been received by the server, but if you never received the acknowledgement, you might try sending it again, and that would create a duplicate.

These are problems that are entirely architected away by using something like the Realm Platform, where you’re guaranteed this full ACID compliance across the network, this full eventual consistency. It’s yet another reason why you should probably be using these objects as messages. Again, if your app were to crash right after sending a message in interprocess communication, the next time it launches, it will still have that locally stored in the Realm, and it can still act on that message.

Q: On the server, for the change notification block on the node side, will an insertion only be called once, even with latency issues?

JP: Correct. Since Realm is fully ACID, not just on the client, but on the server, if for whatever reason the server crashes halfway through trying to process this transaction, the transaction will not complete, because it is atomic. The next time you start it up, you will get that event right back up. It’s basically like a queue system that handles operational transforms and merged conflicts on the fly.