Realm Blog

BYO Authentication: Connect Your Users to the Realm Mobile Platform

by /

We’re making it easier than ever to integrate your existing systems with a modern mobile app, by introducing Custom Authentication on the Realm Mobile Platform. With a few lines of JavaScript, you can bring your own authentication to Realm, allowing your users can log in with their existing credentials and begin using any app built on the Realm Mobile Platform. Using Custom Authentication, you’ll be able to build modern mobile apps with single-sign on (SSO) capabilities, and you can even mobilize your whole legacy backend by integrating our new data connector API, so that your users get access to the data and capabilities they expect.

Legacy systems put you in a double bind when you want to build a new, useful mobile app. If you use your existing backends, it’s likely to be incredibly hard to build the new features your users demand — features that require streaming lots of data from many sources, which tend to overwhelm APIs built for a calmer web. But migrating your API to a more developer-friendly platform is likely to be even harder, exposing you to new security risks in untested authentication APIs, and forcing you to solve problems that you thought you already put behind you, this time in a new language and framework.

Wouldn’t it be great to keep the hard work you put into your API, while also getting all the modern development tools that make it easier to build better apps? That’s what bring-your-own authentication does for the Realm Mobile Platform. You get to integrate Realm with your trusted legacy authentication APIs, so that your users simply bring their username and password (or any other tokens or credentials) to your mobile app, and the Realm Mobile Platform handles the work of authenticating them against your existing API. Your developers don’t just avoid rewriting all their auth code: they also get realtime data sync, offline-first capabilities, and enterprise-grade data connectors for your existing databases and backends — all benefits that you get from building your app on the Realm Mobile Platform.

Here’s what it looks like in practice: you just write one short JavaScript file, then add a line to your configuration.yml file:


module.exports = function(deps) {
  return class FooAuthProvider extends deps.BaseAuthProvider {
    // This is used by the ROS when it loads the available
    // authentication providers. This function is required.
    static get name() {
      return 'custom/fooauth';
    }

    // This is used by ROS when it parses the configuration
    // file, to ensure that required default options are
    // available. This function is optional.
    static get defaultOptions() {
      return {
        auth_server: 'https://my.auth.server.example',
      }
    }

    constructor(name, options, requestPromise) {
      super(name, options, requestPromise);

      // Here you can set your instance variables as usual
      this.httpMethod = 'GET';
    }
  };
}

Now we get to the substance of the authentication provider, where we’ll make a request to your authentication API using the credentials that the user provided:


verifyIdentifier(req) {
  // The token submitted by the client
  const token = req.body.data;
  // The client SDKs also support sending additional information
  // in the form of a map/dictionary. This is accessible through
  // the `req.body.user_info` key.

  // The information required to reach your API server that can
  // validate the token. Obviously, this assumes a REST-like API.
  // Please note that `this.options` comes from the configuration file or
  // `defaultOptions()` method defined above. Similarly, `this.httpMethod` is
  // an instance variable defined in the constructor above.
  const httpOptions = {
    uri: `${this.options.authServer}/api/auth?token=${token}`,
    method: this.httpMethod,
    json: true,
  };

  return this.request(httpOptions)
    .catch((err) => {
      // Please see src/node/services/problem/http_problem.js
      // for other error cases
      throw new problem.HttpProblem.Unauthorized({
        detail: `Something bad happened: ${err.toString()}`,
      });
    })
    .then((result) => {
      // This userID has to be unique, and will be used to either
      // - create a user in the ROS
      // - login the user, if it already exists
      return result.userID;
    });
}

By combining easy authentication with the data connector API, you’ll also be able to connect your apps to the data stored behind your legacy systems. Whether your API is as old as SOAP or just as request-driven as REST, the Realm Mobile Platform’s data connector API will give your users real-time, streaming access to their data they need, and your developers get to use the modern, reactive features of the Realm Mobile Database. You’ll be able to build great app experiences quickly, without trying to teach your older, legacy systems the new tricks they’d need to deal with the scale demands of mobile apps.

To start moving faster on your own mobile journey, just jump into the documentation for custom authentication. If you’d like to learn more about how to completely integrate the Realm Mobile Platform with your legacy databases as well, check out our new PostgreSQL data connector API.

Read more

Making PostgreSQL Realtime

by /

PostgreSQL is one of the most powerful and popular databases in use today, and is increasingly used for systems of record at all types of organizations. This works well for many use cases, but things get difficult when you try to build a modern, reactive app connected to your Postgres database. Postgres is fast, but it’s just not designed for realtime, reactive mobile apps.

Today we’re excited to introduce a solution: Realm’s new PostgreSQL Data Connector. This new connector creates a simple two-way bridge between Postgres and the Realm Mobile Platform, effectively making Postgres realtime for mobile apps. Data flows reliably and in a fault-tolerant manner between Postgres and the Realm Platform, which then handles realtime sync with your mobile apps automatically. This means changes made on mobile clients are automatically reflected back to your Postgres database in realtime, and data changes made in Postgres sync in realtime with client-side databases. You can now easily make truly reactive mobile apps with PostgreSQL and Realm.

Realm Mobile Platform as an alternative to REST APIs

The Realm Mobile Platform replaces traditional methods of using REST APIs to get data in and out of backend systems for use in a mobile context. Our automatic realtime sync service efficiently syncs only what has changed in the data model without the need for manual data requests or bandwidth-hogging polling. Realm automatically handles the realtime sync of your native objects—without the need for JSON—between clients and the Realm Object Server (the server-side component of the Realm Mobile Platform). The Realm Object Server can then provide the API integration point between your mobile application and PostgreSQL database to consolidate API management in the backend, where network issues are not a concern. This architecture reduces the number of API calls that each device must make and reduces the need for complex idempotent APIs that can handle unreliable network connectivity.

PostgreSQL Data Connector Demo

Here’s a video to show you an example of how we can link Postgres and Realm Mobile Platform:

Using sample data from a fictional DVD Rental company, we’re able to modify inventory levels via a mobile app or within Postgres and see those changes sync in realtime. It’s a simple example of how Realm can dramatically simplify making Postgres realtime for your applications.

Realm connects your mobile app and Postgres with automatic realtime sync.

We’re excited to show you how Realm can work for you in your environment. You can visit GitHub for the code and an in-depth overview of how to connect the Realm Mobile Platform and PostgreSQL to create reactive, realtime mobile applications with data stored in PostgreSQL.

Read more

Realm Everywhere, with JavaScript: Universal Node.js Support

by /

JavaScript is probably the hardest working programming language in the world these days, and at Realm, we’re big fans. We love it because it fits great with Realm’s reactive architecture, and because JavaScript is virtually omnipresent. We’ve made it a key part of our platform, so that any developer with JS skills can use Realm. We started by shipping support for the Realm Mobile Database to React Native, and then we brought Realm to Node.js on Linux and macOS, which helps power the Realm Mobile Platform. And today we’re happy to announce Node.js support for Windows, meaning that you can use Realm wherever you can run JavaScript.

Realm now supports not only React Native, but also Node.js on Windows, Linux and macOS. Just like with Realm on mobile, you get all the advantages of the Realm Mobile Database — enabling you to easily and performantly persist any data you need to build your app. We’re also hard at work building support for the Realm Mobile Platform on Win32, so that you’ll be able to sync data with ease and build exciting real-time, collaborative apps using JavaScript.

Realm isn’t just easier to use. It also pairs with JavaScript’s advantages to enable better app experiences. Realm’s reactive principles mean data is automatically refreshed as it changes, making it fit beautifully in an event-driven architecture. It’s also cross-platform, so that you can take advantage of JavaScript’s enormous reach in order to deliver apps to your customers, no matter what devices they use. And finally, our universal support for Node.js means you can even use it for server-side use cases — like using Realm and Node.js to do interprocess communication on a webserver.

We’re making it easier to build the coolest, newest, best apps, on every platform, on any device. Today, we’ve taken another big leap in our journey to being a truly universal data platform, and we can’t wait to see what you build with it. Jump into the Realm JS documentation to get started with the Realm Mobile Database.

Read more

Integrating Azure Authentication with Realm

by /

Today we’re pleased to introduce a new method of client authentication with Azure Active Directory for our community of .NET developers using the Realm Mobile Platform. Now organizations can create reactive apps that work with Microsoft App Service Authentication / Authorization for simplified sign on and identity management.

Setting up Azure Active Directory

Each Azure account has a default AD instance pre-created. For the purposes of this post, we’ll use that one, but you can easily create a separate one by following a tutorial similar to this one.

Once you have the directory setup, create a new user by clicking Add a User in the Quick tasks pane:

add-user

Setting up the Application

To authenticate on the device, we’ll need to setup an Application. Go to App registrations and press Add:

add-app

Specify Name, and set Application Type to Native. The Redirect URI will be used by the client library to identify when the login flow has completed, so it has to be a valid Url, but doesn’t need to be a physical endpoint (as we’ll never load it).

app-details

Once the application is created, take a note of its Application Id, as we’ll need it later.

Authenticating on the client

For the client authentication, we’ll use the Active Directory Authentication Library (ADAL) package. It makes obtaining an access token fairly straightforward:

// Call Login() based on your business logic (e.g. when a user presses a button)

const string ApplicationId = "application-id-from-portal";
const string CommonAuthority = "https://login.windows.net/common";
const string RedirectUri = "redirect-uri-from-portal";

public async Task<User> Login()
{
  var authContext = new AuthenticationContext(CommonAuthority);
  var response = await authContext.AcquireTokenAsync("https://graph.windows.net",
                                                     ApplicationId,
                                                     RedirectUri,
                                                     new PlatformParameters(this));

  // We'll use response.AccessToken later
  return null;
}

The last argument of authContext.AcquireTokenAsync is a platform-specific implementation of IPlatformParameters, so if you’re using a shared project to perform the authentication, you could either obtain it via dependency injection, or add an #if PLATFORM directive. Once Login is called, the user will be presented with a webview where they can enter their credentials:

user-login

After successful authentication, the response object will contain some basic user information as well as an access token, that we’ll use to authenticate against the Realm Object Server.

Integrating with Realm

To authenticate against Realm Object Server, we’ll first need to enable the Azure Active Directory provider. Open up configuration.yml, uncomment the azuread section and fill in the Directory Id, that can be found in the Properties section:

# This enables authentication via an Azure Active Directory access token for a specific app.
azuread:
  # The Directory Id as retrieved from the Active Directory properties in the Azure portal.
  tenant_id: 'active-directory-id'

Now head back to the client application’s Login method to wrap it up:

const string ROSUrl = "http://127.0.0.1"; // Or the address where ROS is hosted

public async Task<User> Login()
{
  // same as above
  // var response = (...);

  var credentials = Credentials.AzureAD(response.AccessToken);
  var user = await User.LoginAsync(credentials, ROSUrl);
  return user;
}

What’s next at Realm

We’re excited to expand our support for the Microsoft ecosystem and our announcement today is one small step on that journey. This year, we’ve announced Realm Xamarin 1.0, Windows Desktop support, and easy ways to get your Realm Object Server (part of the Realm Mobile Platform) up and running on Azure. Stay tuned for more coming in 2017!

Read more

Realm Java 3.0: Collection Notifications, Snapshots and Sorting Across Relationships

by /

We’re releasing version 3.0 of Realm Java today. In this release we have enabled sorting across relationships and we’re giving our live collections RealmResults and RealmList a whole lot more life by adding fine-grained collection notifications, so that your app can respond to elements being added, deleted and modified.

Fine-grained Collection Notifications

Up until now, Realm has provided a single shared RealmChangeListener interface that could be registered on our Realm, RealmObject and RealmResults classes. However that interface came with a limitation. It could not provide information about what had changed, only that something had changed. In many cases that was good enough, but in some cases it was not. The most common example being if you wanted to animate changes in a RecyclerView. Due to Realm’s live-updating objects, the solution up until now have been to use a combination of realm.copyFromRealm() and the DiffUtil class from the Android Support Library. This was hardly ideal.

Today, we are shipping fine-grained collection notifications. It is a new interface that can be registered on RealmResults and will provide information about exactly which objects were added, deleted or changed. This will enable you to just refresh those elements that actually did change, making for a much nicer and responsive UI.

We’ve also updated our Android Adapters library, so the RealmRecyclerViewAdapter will now automatically use the fine-grained notifications for smoother animations. Here’s what it looks like in practice, compared to using RecyclerView without collection notifications:

private final OrderedRealmCollectionChangeListener<RealmResults<Person>> changeListener = new OrderedRealmCollectionChangeListener() {
    @Override
    public void onChange(RealmResults<Person> collection, OrderedCollectionChangeSet changeSet) {
        // `null`  means the async query returns the first time.
        if (changeSet == null) {
            notifyDataSetChanged();
            return;
        }
        // For deletions, the adapter has to be notified in reverse order.
        OrderedCollectionChangeSet.Range[] deletions = changeSet.getDeletionRanges();
        for (int i = deletions.length - 1; i >= 0; i--) {
            OrderedCollectionChangeSet.Range range = deletions[i];
            notifyItemRangeRemoved(range.startIndex, range.length);
        }

        OrderedCollectionChangeSet.Range[] insertions = changeSet.getInsertionRanges();
        for (OrderedCollectionChangeSet.Range range : insertions) {
            notifyItemRangeInserted(range.startIndex, range.length);
        }

        OrderedCollectionChangeSet.Range[] modifications = changeSet.getChangeRanges();
        for (OrderedCollectionChangeSet.Range range : modifications) {
            notifyItemRangeChanged(range.startIndex, range.length);
        }
    }
};

Fine-grained collection notifications work just like all other Realm listeners, so if a query was executed asynchronously (for example, by using findAllAsync()), then the fine-grained changeset will also be calculated on a background thread before delivering the result to the UI thread, without the memory overhead of using DiffUtil to calculate what’s changed.

Collection Snapshots

One of the key design principles for Realm is our concept of “live” objects and collections. It means that if you create a RealmResults from a query, it will stay up-to-date whenever the underlying data changes. This is very effective in any reactive architecture, where you can just register a RealmChangeListener and be notified when the query result is updated.

However in some cases, this “live” nature can lead to unexpected problems when looping over a live collection. Take this code for example:

RealmResults<Guest> uninvitedGuests = realm.where(Guests.class)
  .equalTo("inviteSent", false)
  .findAll(); 

for (i = 0; i < uninvitedGuests.size(); i++) {
  realm.beginTransaction()
  uninvitedGuests.get(i).sendInvite();
  realm.commitTransaction();
}

Here, you make a query for all guests not invited, as you want to send an invite to them. If uninvitedGuests were a normal ArrayList, this would work as you expect and all guests would receive an invite.

However because RealmResults are live updated, and because you execute transactions inside the for-loop, Realm will automatically update the RealmResults after each iteration. This would in turn remove the guest from the uninvitedGuests list causing the entire array to shift, but because it is the for-loop keeping track of the current index, this would now be at the wrong position. The result was that only half the guests would receive an invite. Clearly not ideal.

Two solutions existed:

// 1. Wrap the entire loop in a transaction
realm.beginTransaction()
for (i = 0; i < uninvitedGuests.size(); i++) {
  uninvitedGuests.get(i).sendInvite();
}
realm.commitTransaction();

// 2. Loop the list backwards
for (int i = uninvitedGuests.size() - 1; i >= 0; i--) {
  realm.beginTransaction()
  uninvitedGuests.get(i).sendInvite();
  realm.commitTransaction();
}

Neither of these solutions were obvious though, so in Realm Java 0.89 we made the decision to defer all updates of RealmResults to the next looper event (by using Handler.postAtFrontOfQueue).

This had the advantage that the simple for-loop would work as expected, but it came with the price that the RealmResults could now get a little out of sync, e.g. if we deleted the guest objects instead of setting a boolean on them, they would still be present in the query result, but just as invalid objects.

realm.beginTransaction();
guests.get(0).deleteFromRealm(); // Delete the object from Realm
realm.commitTransaction();
guests.get(0).isValid() == false; // You could now get a reference to a deleted object.

This would rarely happen in practise though, so you could say we chose standard iterator behavior over staying true to core Realm architectural decisions.

In 3.0 we are revisiting this decision by making RealmResults fully live again, but to make it easier to work with live collections in simple for-loops, we’re adding a createSnapshot() method that can provide the current stable list. Simple for-loops should now use this snapshot if they intend to modify data in a loop (note that in most cases doing transactions in a loop is still an anti-pattern).

RealmResults<Person> uninvitedGuests = realm.where(Person.class).equalTo("inviteSent", false).findAll();
OrderedRealmCollectionSnapshot<Person> uninvitedGuestsSnapshot = uninvitedGuests.createSnapshot();
for (int i = 0; uninvitedGuestsSnapshot.size(); i++) {
    realm.beginTransaction();
    uninvitedGuestsSnapshot.get(i).setInvited(true);
    realm.commitTransaction();
}

Iterators will use this snapshot behind the scenes, so they continue to work in the same way you expect them to today:

realm.beginTransaction();
RealmResults<Person> uninvitedGuests = realm.where(Person.class).equalTo("inviteSent", false).findAll();
for (Person guest : guests) {
    realm.beginTransaction();
    uninvitedGuests.setInvited(true); 
    realm.commitTransaction();
}

We are doing this for a number of reasons. One of them being internal refactorings needed to support fine-grained collection notifications. The other is that it means the semantics of Realm classes are all the same since both RealmResults and RealmList are fully live, and it is easier to both reason about them and document their usage. This had caused some confusion in the past, but we believe that fully supporting and documenting the live nature of Realm classes means we can offer users the most powerful APIs.

We are aware that this change might cause some headaches in existing code bases, but this change will only affect your work if you’re performing write transactions inside simple for-loops. In every other case, your code will continue to work as normal, and we hope that you agree it is better in the long term.

Sorting across relationships

Until now, it was only possible to sort RealmResults using “direct” properties. Starting with Realm Java 3.0, you can now sort query results by properties on the other side of an objects many-to-one relationships

For example, to sort a collection of Person objects based on their dog’s age, you can now do the following:

RealmResults<Person> persons = realm.where(Person.class).findAllSorted(dog.age, Sort.ASCENDING);

We’ve also shipped a bunch of bug fixes in the last couple releases. To see the full list of changes, check out the changelog.


Thanks for reading. Now go forth and build amazing apps with Realm! As always, we’re around on Stack Overflow, GitHub, or Twitter.

Read more

Realm + Microsoft: Xamarin, Azure, and Windows Desktop

by /

Realm’s mission is to make it easier to build great apps, and we want to bring that reality to as many developers as possible. So today, we’re making it easy for Microsoft developers to use Realm to build highly-scalable, production-ready realtime apps. We’ve added new Realm Mobile Database features to Xamarin, introduced Xamarin support to the Realm Mobile Platform, and even brought the Realm Mobile Database to the Windows Desktop platform.

We’ve worked with Microsoft’s Xamarin team to build a sample app and tutorial using Xamarin and Azure that will show you just how powerful all these new tools are. Read on to see how Realm and Microsoft fit together.

Xamarin and Realm: Introducing Xamarin to the Realm Mobile Platform, plus new Realm Mobile Database features

We’ve been hard at work on Realm Xamarin — our entirely free, entirely open source client-side object database — and after the past several months shipping property (object-level) notifications, backlink support and testing, testing, testing, we’re officially releasing Realm Xamarin 1.0, and declaring it ready for production. We’re grateful to the community of developers who have helped us get here — 521 closed issues later.

But that’s not all we’ve got for Xamarin. Starting today, the Realm Mobile Platform supports Xamarin. Now, C# developers can build cross-platform reactive apps simply by connecting a Realm-backed app to the Realm Object Server. Without writing any serialization or networking code, the Realm Mobile Platform gives you everything you need to build rich, realtime experiences, by handling conflict resolution, user authentication, and customizable permissions. And because you’ve got the Realm Mobile Database storing data on the client, your users also get a first-class offline experience.

Curious what all those features add up to? Well, here’s what it looks like:

To see how easy it is to build the Draw app on your own, check out the tutorial we collaborated on with Microsoft’s Xamarin team. By the end of it, you’ll have a realtime cross-platform app built on Xamarin, syncing with a Realm Object Server deployed on Azure. You can also join our webinar on March 9th to follow along in realtime as we go through it.

If you’re already familiar with the Realm Mobile Platform, open our Realm Object Server documentation, and you can get your apps up and syncing with a single click and a few lines of code.

Realm Database Support for Windows Desktop

Finally, we’re also releasing support for the Realm Mobile Database on Windows Desktop, allowing you to run Realm on the hundreds of millions of computers that support the Win32 API. With this new release, developers receive the same benefits from using an object database and Realm’s APIs as our mobile developer communities enjoy, enabling Realm-backed Windows desktop applications and the ability to create custom tooling that can generate and view Realm data. You simply define a schema, and start using the objects you’re used to. When you’re ready to persist those objects to disk, you just include them in a write transaction, and they’re preserved in our ACID-compliant database until you need to use them again. Check out our documentation to find installation instructions, and get building today.

Start Building with Realm and Microsoft

We’re thrilled to bring new and exciting tools to such an enormous, thriving community of developers, and we can’t wait to see what you’ll build with Realm, Xamarin, and Microsoft. It’s as easy as jumping into the Realm Xamarin documentation. And if you’d like to see the new Microsoft tooling and features in action, join us in a technical webinar this Thursday, March 9th.

Read more

Realm React Native 1.0: Realm Mobile Platform Support

by /

Today, we’ve got two announcements for the React Native community. First, after nearly a year of open source collaboration, Realm React Native has reached 1.0, marking a key milestone as a powerful object database, and a foundation for great reactive apps. We’re also very excited to announce that the Realm Mobile Platform now supports React Native. With just a few lines of code, React Native developers can now build exciting new experiences on top of our realtime platform.

When we launched the beta of Realm React Native early last year, we found lots of uptake from React Native’s passionate developer community, and from companies like TaskRabbit. Because Realm is a cross-platform database, you can build apps that reuse more code, making your data layer more maintainable and less fragile. And Realm’s live objects and collection notifications mean that your app really is reactive — it responds in realtime as the Realm-backed data updates. To see how easy it is to get started, just check out our Realm React Native documentation.

But Realm is more than just a client-side database. With the release of Realm React Native 1.0, React Native developers can now build powerful new apps on top of the Realm Mobile Platform. By connecting to the Realm Object Server and then creating synced Realms in your app, your Realm-backed objects will automatically sync whenever their properties change, while still being stored locally for offline use. The Realm Mobile Platform also gives you tools for handling conflict resolution, user authentication, and customizable permissions, so that you have all you need to build compelling, collaborative experiences.

The Professional and Enterprise Editions also include Event Handling, which lets you write reactive server-side JavaScript code. Now, you can bring reactive principles across your whole codebase, ensuring that your app responds predictably whenever and wherever user data changes.

It’s easy to integrate your own React Native app with the Realm Mobile Platform. Authenticate a user, connect to the Realm Object Server, then reactively update your app’s UI with this code sample:


import React, { Component } from 'react';
import { Text } from 'react-native';
import Realm from 'realm';
import { ListView } from 'realm/react-native';

export default class DogsView extends Component {
    constructor(props) {
        super(props);
        // Initialize the component with an empty data source
        const ds = new ListView.DataSource({rowHasChanged: (r1, r2) => r1 !== r2});
        this.state = { dataSource: ds };
    }

    componentWillMount() {
        // `this.props` contains the username and password passed to the DogsView instance
        Realm.Sync.User.login('https://my-realm-server.com', this.props.username, this.props.password, (error, user) => {
            let realm = new Realm({
                schema: [ { name: 'Dog', properties: { name: 'string' } } ],
                sync: { user, url: 'realms://my-realm-server.com/~/dogs'}
            });
            // Once the user is logged in and we have a synced realm,
            // reset the DataSource with all the Dog objects in the realm.
            // Also subscribe for changes on the Dogs collection and refresh the UI when they occur
			const dogs = realm.objects('Dog');
            this.setState({ realm, dataSource: this.state.dataSource.cloneWithRows(dogs) });
			dogs.addListener(() => this.forceUpdate());
        });
    }

    render() {
        return (<ListView dataSource={this.state.dataSource} renderRow={(item) => <Text>{item.name}</Text>} />);
    }
}

To get started with the Realm Mobile Platform, check out our documentation. Start making something your users will really love — across all their devices, whether they’re online or offline.

Read more

Realm ObjC & Swift 2.4: Object Notifications!

by /

We’re releasing version 2.4 of Realm Objective‑C and Realm Swift today. In this release, we’re giving our live objects a whole lot more life by adding object-level notifications, so that your app can respond as individual objects update.

Object-Level Notifications

Up until now, if you wanted to know that an object changed, you had to get notifications from collections, which tell you when objects contained in that collection get deleted, inserted, or modified. However, you wouldn’t know which properties have changed. Additionally, creating collections in order to asynchronously observe a single object is awkward, costly and difficult to define when an object can’t easily be uniquely represented by a query.

We’ve long supported KVO to observe property-level changes to single objects, but it’s no secret that KVO is pretty gross, you can only observe a single property per KVO notification, and KVO isn’t cross-platform or cross-language. KVO notifications are also run synchronously, which often means blocking your UI thread 🙀.

Today, we’re shipping object-level notifications so that you’re notified of changes on a single object in the most helpful way possible. Object notifications are powered by a very simple API. Like with collection notifications, you simply add a notification block to the object you’d like to observe. Creating the notification will return a NotificationToken; hold onto it as long as you need notifications for the object.

When the notifications start rolling in, the callback lets you see exactly what changed with its ObjectChange struct, which will expose exactly what properties have changed, or whether the observed object has been deleted:

class Message: Object {
  dynamic var isRead = false
  dynamic var content = ""
}

class MessageController: UIViewController {
  private let notificationToken: NotificationToken

  init(message: Message) {
    notificationToken = message.addNotificationBlock { change in
      switch change {
      case .change(let properties):
        if let readChange = properties.first(where: { $0.name == "isRead" }) {
          self.showReadBadge = readChange.newValue as! Bool
        }
        if let contentChange = properties.first(where: { $0.name == "content" }) {
          self.contentView.textValue = contentChange.newValue as! String
        }
      case .deleted:
        self.handleDeletion()
      case .error(let error):
        self.handleError(error)
      }
    }
  }

  deinit {
    notificationToken.stop()
  }
}

When you’re ready to stop watching the object, you call token.stop(), or simply release the token object.

In summary, you create a notification on a managed object with addNotificationBlock, hold onto the returned NotificationToken instance, and handle whatever updates you need to make to your app inside the callback.

A couple caveats: when list properties change, you won’t see newValue or oldValue data, but the notification will still fire. And if you’re in need of making a synchronous UI write, or just need to skip the notification for a given write, you can still use the interface-driven writes feature by passing the object’s notification token to realm.commitWrite(withoutNotifying: [token]).

Bug Fixes

We’ve fixed several error handling issues when renewing expired authentication tokens for synchronized Realms in this release.

We strongly encourage all Realm Mobile Platform users to update to this version to avoid issues with expiring tokens.

Oh, and we also managed to fix an elusive bug causing a “bad transaction log” exception that’s been haunting us for months! #RaceConditions🏎

Deprecating Swift 2

Supporting legacy Swift versions is a lot of work 😅. With less than 5% of active Realm builders on Swift 2.x and a high cost of maintenance for Swift 2 on our end, this release drops support for Swift 2.x.

That makes Realm Swift 2.3.0 is the last version to support Swift 2.x. We apologize for the inconvenience.

A Note About Xcode 8.3 Beta 1

Apple published the first Xcode 8.3 beta earlier this week. We’ve included a change in this release that fixes a compilation issue with Realm Objective-C.

However, we’re still working through Swift 3.1 compilation issues. Please subscribe to #4586 to be notified of any progress to support Xcode 8.3 with Realm Swift.


Thanks for reading. Now go forth and build amazing apps with Realm! As always, we’re around on Stack Overflow, GitHub, or Twitter.

Read more

The Platform Experience: Announcing Realm World Tour

by /

It’s time for us to hit the road! We just shipped Realm Mobile Platform 1.0, so now we’re sending our top product engineers out to events in 20+ cities across three continents, to bring the Realm platform to you live and in realtime ;-) It’s the first-ever Realm World Tour, and we’re excited to get the chance to meet you, to help you get started with Realm, and to hear about what you are working on!

We’ve reached out to leaders of the developer communities at each city, and with their help as local hosts, have planned a great evening for attendees. Each event includes an introduction to the free Developer Edition plus discussion of some of the advanced features, followed by sample code, practical demos, and extensive technical Q&A session — all led by a Realm product engineer. There will also be plenty of time to talk one-to-one, to share best practices, and to learn from each other. And, we’ll have some special attendees-only swag to give out – we’re keeping the design a secret until the tour kick-off in our HQ in San Francisco on February 15!

The Realm World Tour events are free, but space is limited — check out the official web page for the latest info and chance to register.

Read more

Ready for Realtime and Scale: Announcing Realm Mobile Platform 1.0

by /

Today we’re very proud to announce that Realm Mobile Platform 1.0 is now generally available and ready for production use cases ranging from the very smallest to those with massive scale. We’ve been hard at work over the last few months fixing bugs, polishing rough edges, and hardening the code, as well as adding more than a few new features (more on those below). Hitting 1.0 is a huge milestone for all of us at Realm, and we are extremely grateful to the customers and members of the community who have helped us get here with their feedback throughout the beta period!

The Realm Mobile Platform makes it easier for developers to build apps with sophisticated features like realtime collaboration, live messaging, robust offline-first experiences, and more. The platform combines Realm’s popular client-side databases for iOS and Android with the new Realm Object Server, and delivers automatic realtime data synchronization. As a developer, you don’t need to write or maintain networking code or do any serialization, Realm handles the sync for you — you can be sure that the data on the device is always synchronized with the data on the server. Realm also makes it easy to create server-side logic to react instantly to changes in data, and build integrations to existing systems and APIs.

With this release, all three editions of the platform are ready for you to start building, and we have some fun new demo code you can run to try it out. Read on for more of the details… or, if you’d rather just dive in, just download our Draw demo app and the free Developer Edition for Mac or Linux and start exploring.

Three New Enterprise Features

This release also brings three new features to the Enterprise Edition:

Data Integration API

Our new Data Integration API allows developers to easily integrate the Realm Mobile Platform with any data source, making it simple to use Realm for connecting realtime apps and app features to existing data and services. Many of our beta users are leveraging this new Realm feature to “mobilize” their legacy systems, using Realm as a realtime bridge that gives their Android and iOS teams an easy way to build modern, reactive apps connected to data and logic in existing systems of record.

For example, imagine a retailer who wants to launch a new mobile shopping experience, but needs that new app to be seamlessly connected to an existing payment transaction database. Using Realm’s new Data Integration API, Realm is able to reliably connect the existing database to Realm Object Server such that the Object Server and the payment transaction database are synchronized with the same data. To ensure data is accurate in all locations, the API is designed to understand broken connections and restart interrupted transactions. With these guarantees in place, Realm Object Server can now function as middleware between the client-side databases and the payment transaction database, allowing the retailer to leverage their existing database investments with Realm for new projects.

The Data Integration API makes it easy to build your own custom connectors, but we’re also announcing the first of our pre-built connectors, for PostgreSQL. We’re working on others, and based on customer requests expect to soon release pre-built connectors for Oracle, MongoDB, Hadoop, SAP HANA, and Redis.

Horizontal Scalability

We’ve built the Realm platform to be quite robust – depending on the app and the hardware you run it on, a single Realm Object Server can handle well over ten thousand concurrent users. But to allow customers to reach massive scale, we’ve added horizontal scaling to the Enterprise Edition. Included, is a built-in load balancer that automatically handles distributing connections. This makes it possible to deploy multiple instances of Realm Object Server in parallel, so your app can support more than one million concurrent realtime users.

Continuous Backups

Continuous Backups allow you to create automatic backups of your production Realm Object Server data on a secondary server. In the event of disaster (power outage, hardware failure, natural disaster, etc.), you can continue to operate your application from the secondary server. Developers can choose where to locate their secondary backup servers, including a different region, for increased disaster recovery resiliency.

New Features in Enterprise Edition

Getting Started with Realm and the Draw Demo App

Your first step is to install the Realm Platform, on your own machine or on the cloud service of your choice. Unless you have a specific interest in exploring event handling or other Pro Edition features, we recommend you start with the Developer Edition — it is powerful and entirely free of cost, with no time limits, and you can use it for everything from hacking around to live production apps (see the license terms here). Our documentation will walk you through install, and get you up and running.

Once you have Realm installed, you’re ready to run some sample code. Draw is a whiteboard collaboration app that showcases Realm handling realtime data synchronization between multiple users. It’s totally open source, and available along with other Realm demos on GitHub. This video of Draw in action shows the realtime data synchronization between two iPads, and what happens when the cell network temporarily fails.

How to Learn More

If you’d like to get an in-depth walkthrough of the Realm Mobile Platform and a chance to ask questions, our product team will be doing a live technical session. Please register here.

Read more

Realm ObjC & Swift 2.3: Sync Progress Notifications, Improved Sharing & Backup Recovery!

by /

We’re releasing version 2.3 of Realm Objective‑C and Realm Swift today, which includes several improvements to the Realm Mobile Platform features, such as sync progress notifications, backup recovery, and more flexible sharing mechanisms. Read on to learn more!

Sync Progress Notifications

Realm Mobile Platform offers a true offline-first experience, where changes can be applied immediately to the local Realm irrelevant of the network conditions. In the background, the synchronization happens automatically, and when the network is available, changes quickly propagate across devices and the server. This enables the realtime, collaborative experiences unique to Realm.

There are times, however, when it’s useful to know how much data is left to transfer. Perhaps on the first launch of your app, you’d like to make sure a sizable amount of the data is available before presenting your user interface. At other times, your app would simply benefit from showing when data is synchronizing, by presenting a progress bar or activity indicator.

This is why we’ve added APIs to monitor sync progress by registering notification blocks on SyncSession. Simply, specify the transfer direction (.upload/.download) and mode (.reportIndefinitely/.forCurrentlyOutstandingWork) to monitor.

For example, to show an upload progress bar after writing a large image to your Realm and waiting for it to upload, you might write this:

let session = SyncUser.current.session(for: realmURL)!
self.token = session.addProgressNotification(for: .upload,
                                             mode: .forCurrentlyOutstandingWork) { progress in
  self.updateProgressBar(fraction: progress.fractionTransferred)
  if progress.isTransferComplete {
    self.hideProgressBar()
    self.token.stop()
  }
}

Or to show an activity indicator for the entire lifetime of your app, you might write this:

let session = SyncUser.current.session(for: realmURL)!
self.token = session.addProgressNotification(for: .download,
                                             mode: .reportIndefinitely) { progress in
  if progress.isTransferComplete {
    self.hideActivityIndicator()
  } else {
    self.showActivityIndicator()
  }
}

Flexible Sharing Mechanism

Realtime collaboration and data sharing is an important pillar of the Realm Mobile Platform. To enable this, we made it possible for mobile developers to share Realms between users when we introduced the PermissionChange APIs in November.

Today, we’re making it even easier to share Realms between users, allowing extremely flexible sharing and permission management, all controlled from the Realm client APIs and without the need to write any server code! Specifically, we’re adding the SyncPermissionOffer and SyncPermissionOfferResponse classes to allow creating and accepting permission change events for synchronized Realms between different users.

These APIs once again demonstrate the power of the Realm Mobile Platform’s objects-as-APIs approach. Making and receiving permission offers is very similar to the existing PermissionChange process. Simply create the object in the current user’s management Realm and observe it to know when it was synchronized and processed by the Realm Object Server.

Sharing a synchronized Realm is as easy as following these steps:

  1. Create a SyncPermissionOffer object in the user’s management Realm.
  2. Wait for the offer to be synced and processed by the server.
  3. Once the token property of the object is populated, send it to another user however you’d like: email, iMessage, Action Controller, carrier pigeon, whatever you like!
  4. The receiving user then creates a SyncPermissionOfferResponse object in his/her management Realm.
  5. The receiving user then waits for the response to be synced and processed by the server.
  6. Once the response has been processed, the receiving user can now access the Realm at the response’s realmUrl property.

For example:

////////////////
// Sender
////////////////

// Create offer with full permissions
let shareOffer = SyncPermissionOffer(realmURL: realmURL, expiresAt: nil,
                                     mayRead: true, mayWrite: true, mayManage: true)
// Add to management Realm to sync with ROS
try managementRealm.write {
  managementRealm.add(shareOffer)
}
// Wait for server to process
let offerResults = managementRealm.objects(SyncPermissionOffer.self).filter("id = %@", shareOffer.id)
shareOfferNotificationToken = offerResults.addNotificationBlock { _ in
  guard case let offer = offerResults.first,
             offer.status == .success,
             let token = offer.token else {
    return
  }
  // Send token via UIActivityViewController
  let url = "realmtasks://" + token.replacingOccurrences(of: ":", with: "/")
  let activityViewController = UIActivityViewController(activityItems: [url], applicationActivities: nil)
  self.present(activityViewController, animated: true, completion: nil)
}

////////////////
// Receiver
////////////////

// Create response with received token
let response = SyncPermissionOfferResponse(token: token)
try managementRealm.write {
  managementRealm.add(response)
}
// Wait for server to process
let responseResults = managementRealm.objects(SyncPermissionOfferResponse.self).filter("id = %@", response.id)
acceptShareNotificationToken = responseResults.addNotificationBlock { _ in
  guard case let response = responseResults.first,
             response.status == .success,
             let realmURL = response.realmUrl else {
    return
  }
  // User can now access Realm at realmURL 🎉
}

We have a Proof-Of-Concept branch towards RealmTasks that demonstrates how sharing lists between users could be built using this mechanism.

Backup Recovery

When your servers go down, you need a plan to recover. That’s why the Realm Mobile Platform has offered the ability to back up your data for a few months now. Today we’re announcing our Continuous Backup solution to automate keeping up-to-date backups.

Under normal conditions, Realm’s synchronization engine works by transferring just the specific operations. When the Realm Object Server confirms the receipt of new operations, the local logs are cleaned up. This helps keep your app’s disk usage small and Realm blazing fast ⚡️.

Furthermore, because Realm is an offline-first database, if your Realm Object Server is down for whatever reason, all your local data stays available.

However, if you need to recover from a backup on your server, your clients will receive a “client reset” error via the global error handler. Note that you may continue to use the local Realm as you normally would, but that any subsequent changes, or changes made after the last backup point, will be lost.

Once you receive a “client reset” error, you could inform the user about the situation, stop accessing the Realm and redownload the Realm from the server at the latest backed up version. The NSError’s userInfo property contains a block that you may call to clean up the previous local version of the Realm.

If you choose to not handle the client reset error immediately, the next time your app launches, the previous local version of the Realm will be deleted and redownloaded from the server at the latest backed up version automatically on first access.

Bug Fixes

  • Fix a call to commitWrite(withoutNotifying:) committing a transaction that would not have triggered a notification incorrectly skipping the next notification.
  • Fix incorrect results and crashes when conflicting object insertions are merged by the synchronization mechanism when there is a collection notification registered for that object type.

Legacy Swift Version Support

We’d like to remind you that we will continue to support Xcode 7.3.1 and Swift 2.x as long as we can, but encourage all our users to migrate to Swift 3 as soon as possible.


Thanks for reading. Now go forth and build amazing apps with Realm! As always, we’re around on Stack Overflow, GitHub, or Twitter.

Read more

Realm Java 2.3: Improved Sharing, Backup Recovery and Wildcard Queries

by /

We just released a new version of Realm Java to this website and to Bintray. It contains a new wildcard query syntax as well as several improvements to the Realm Mobile Platform such as backup recovery and a more flexible data sharing and permission management system. Read on to learn more.

Flexible Data Sharing

Realtime collaboration and data sharing is an important pillar of the Realm Mobile Platform. To enable this, we made it possible for mobile developers to share Realms between users when we introduced the PermissionChange APIs in November.

Today, we’re making it even easier to share Realms between users, allowing extremely flexible sharing and permission management, all controlled from the Realm client APIs and without the need to write any server code! Specifically, we’re adding the PermissionOffer and PermissionOfferResponse classes to allow creating and accepting permission change events for synchronized Realms between different users.

These APIs once again demonstrate the power of the Realm Mobile Platform’s objects-as-APIs approach. Making and receiving permission offers is very similar to the existing PermissionChange process. Simply create the object in the current user’s management Realm and observe it to know when it was synchronized and processed by the Realm Object Server.

Sharing a synchronized Realm is as easy as following these steps:

  1. Create a PermissionOffer object in the user’s management Realm.
  2. Wait for a notification that the token of the object is populated by the server. Once the token property of the object is populated, send it to another user however you’d like: push notification, email, carrier pigeon!
  3. The receiving user then creates a PermissionOfferResponse object in his/her management Realm.
  4. The receiving user then waits for the response to be synced and processed by the server.
  5. Once the response has been processed, the receiving user can now access the Realm at the response’s realmUrl property.

For example:

/**
 * Sender
 */
SyncUser user = getUser("user");
Realm managementRealm = user.getManagementRealm();
String sharedRealmUrl = "realm://my.server/~/my-realm";
boolean mayRead = true;
boolean mayWrite = true;
boolean mayManage = true;
Date expiresAt = null; // Offer never expires;
final PermissionOffer offer = new PermissionOffer(sharedRealmUrl, mayRead, mayWrite, mayManage, expiresAt);
String offerId = offer.getId();
managementRealm.executeTransaction(new Realm.Transaction() {
    @Override
    public void execute(Realm realm) {
        realm.insert(offer);
    }
});

// Wait for server to handle the offer
RealmResults<PermissionOffer> offers = managementRealm.where(PermissionOffer.class)
        .equalTo("id", offerId)
        .equalTo("statusCode", 0)
        .findAll();
offers.addChangeListener(new RealmChangeListener<RealmResults<PermissionOffer>>() {
    @Override
    public void onChange(RealmResults<PermissionOffer> offers) {
        PermissionOffer offer = offers.first();
        String token = offer.getToken();
        // Offer is ready, send token to the other user
        sendTokenToOtherUser(token);
    }
});

/**
 * Receiver
 */
final SyncUser user2 = getUser("user2");
Realm managementRealm = user.getManagementRealm();

// Accept the offer
String offerToken = getToken();
final PermissionOfferResponse offerResponse = new PermissionOfferResponse(offerToken);
managementRealm.executeTransaction(new Realm.Transaction() {
    @Override
    public void execute(Realm realm) {
        realm.insert(offerResponse);
    }
});

We have a Proof-Of-Concept branch towards RealmTasks that demonstrates how sharing lists between users could be built using this mechanism.

Backup Recovery

When your servers goes down, you need a plan to recover. That’s why the Realm Mobile Platform has offered the ability to back up your data for a few months now. Today we’re announcing our Continuous Backup solution to automate keeping up-to-date backups.

Under normal conditions, Realm’s synchronization engine works by transferring just the specific operations. When the Realm Object Server confirms the receipt of new operations, the local logs are cleaned up. This helps keep your app’s disk usage small and Realm blazing fast ⚡️.

Furthermore, because Realm is an offline-first database, if your Realm Object Server is down for whatever reason, all your local data stays available.

However, if you need to recover from a backup on your server, your clients will receive a “client reset” error via the global error handler. Note that you may continue to use the local Realm as you normally would, but that any subsequent changes, or changes made after the last backup point, will be lost.

The next time your app launches, the previous local version of the Realm will be deleted and redownloaded from the server at the latest backed up version automatically on first access.

Wildcard Queries

Until now, the string query predicates have been limited to substring searches. With this release, we introduce the like predicate which can do simple pattern matching. ? will match a single-character while * matches zero or more characters. For example

String[] names = {"Jim", "Simon", "Jimmy", "Tommy", "Jamie"};

realm.beginTransaction();
for (String name : names) {
    Person person = realm.createObject(Person.class);
    person.setName(name);
}
realm.commitTransaction();
RealmResults<Person> persons1 = realm.where(Person.class).like("name", "J?m*").findAll(); // => Jim, Jimmy, Jamie
RealmResults<Person> persons2 = realm.where(Person.class).like("name", "?im*").findAll(); // => Jim, Simon, Jimmy

Please read the API documentation to get the full picture on how to use like.

Bug Fixes

  • Fixed native memory leak setting the value of a primary key.
  • Fixed “too many open files” issue.
  • Activated Realm’s annotation processor on connectedTest when the project is using kapt.

Thanks for reading. Now go forth and build amazing apps with Realm! As always, we’re around on Stack Overflow, GitHub, or Twitter.

Read more

JP Simard on Realm & Open Source on the Consult Podcast

by /

Consult is a monthly interview podcast hosted by David Kopec. It focuses on software developers who work on Apple platforms to create client products.

JP Simard, iOS lead at Realm, joins the show to discuss the Realm Mobile Database and the new Realm Mobile Platform. JP digs into the technical details that make Realm a great choice for developers. He also discusses his prodigious open source work and his prior life as lead of an iOS consulting firm.

Listen to the full interview here, or check out the episode’s transcript below.


Introduction (0:00)

Hi everyone. Welcome to the November 2016 episode of Consult. I do want to reach out to the community, so if you know somebody who you think would be great on Consult, let me know. I’m @davekopec on Twitter.

My interview this month is with JP Simard of Realm. You all know Realm, the popular, open source mobile data base that’s really starting to take on Core Data. We really dig into the details of Realm and we also get into JP’s previous career as an iOS consultant.

A lot of you probably know JP from the open source community. He’s also the iOS lead at Realm.

Background (1:58)

Q: Tell us a little bit about your background. Go way back. How did you first get into computing? I know you had a consulting company for a number of years, tell us about that too.

JP: Going way back for me doesn’t actually go back all that far. My first intro into programming was in university. I was taking electrical engineering and I was honestly pretty fed up with the very long project cycles. You send something over to fab, or you do simulations. Essentially, every project either takes something on the scale of months or years, if you look at commercial projects. That just wasn’t very exciting for me.

I ended up building this technology that was linked to electrical engineering. Then I wanted to build an app around it. I stared programming, trying to figure out how you actually do that and then I realized that I had to pay the bills. I went around a few different festivals in my hometown in Ottawa, Canada. I tried to pitch them on making an app, even though I had never done it before. Eventually one of them stuck, and ended up having to build an app for them. This was in 2011, right when I was getting started.

I got by right by the skin of my teeth. I managed to patch together just enough of an app and interface builder to satisfy the contract and move on. I continued with that process, going door to door, asking friends, acquaintances, trying to figure out who needed an app, saying that I could build it without ever having anything close to it. Over time I ended up actually fulfilling those contracts, then having more contracts and growing that team.

Essentially, that became a business that was initially spawned by this idea of just scratching my own itch, making my own app. Ultimately, I never ended up building that app, but did grow that agency, something called Magnetic Bear Studios into a small team. It consisted of a few developers, a few designers and making apps for all sorts of companies across North America. At that point I was really hooked into programming. There was no escape.

iOS in Canada (4:29)

Q: Very interesting. Tell us a little bit about the iOS scene in Canada, in Ottawa more specifically. I actually lived for Ottawa for a year when I was a little kid, but I know nothing about the iOS scene there. What’s it like?

JP: I love it because it’s a very tight knit community. Now living in Silicon Valley, in San Francisco, there are definitely a lot of people in the community here. You tend to go to meet-ups. There are tons of them to start off with, and they’re actually competing in some sense; competing for the market share of local developers. Not only that, you tend to have such large groups or groups with members that pop in and out every now and again. There’s a really big difference between that, and say specifically, the Apple developer community in a small-ish city like Ottawa where you have people who are really ingrained in the community.

It’s a very small, tight knit group, a group of maybe 20 or 30 really hardcore people who have been doing Mac programming since the 90s, for some of them. People who have really been ingrained in the industry. There are so few of them there, relatively speaking, compared to a major hub like New York City, London, or San Francisco, that you really do have to stick together. It’s very regular meetings for groups like Cocoa Heads, for example, and it’s people who are keen on sharing ideas and sharing what they learn on a very regular basis and keeping in touch.

It’s an interesting dynamic, and I certainly miss it in a lot of ways. I like being nostalgic every now and again, thinking back to those days. In the last few years, there’s this conference called NSNorth that’s run by Philippe Casgrain and Dan Byers. They’re able to attract people from all around the world to come to this conference because they’re so passionate about it and because they’ve been doing it for decades at this point, and the community is really small but strong.

Clients (7:00)

Q: I’ve always heard tremendous things about NSNorth and what a great experience it is for the attendees. When you were at Magnetic Bear Studios and running that, did you have mostly local clients or were the clients from all across Canada and the United States?

JP: For the most part they were local clients. We did have a handful of clients when we were growing and making a name for ourselves that were everywhere from Silicon Valley to both coasts of Canada. Ultimately, what lead to the end of the Magnetic Bear Studios agency was really my own limitations. At that point, I was running this business that I had no idea how to run. It was mostly focused on hustling and doing sales, doing business development, doing everything from accounting to everything that falls into managing small business.

Ultimately, not only was I not very good at that, I also didn’t really like it. All of that lead to the conclusion that towards the end of that three and a half year stint I was really looking for something a little bit more technical to sink my teeth into and that’s when I joined Realm.

Building a Team (8:24)

Q: I want to get more into Realm, but still talking about Magnetic Bear Studios, what was it like building out that team in Canada? Was there enough iOS local talent there? What’s the supply and demand like for iOS labor in Canada?

JP: Answering that in the present tense is really different than the landscape as it was in 2011-2013 when I was really involved. Back then, people were still riding the mobile wave as the hot new thing. Today you look at buzz words and they’ve shifted. Even though mobile is now a concrete part of a lot of businesses’ strategies, back in 2011 and 2012 people knew it was going to be big, it was already established at that point. The iPhone had been out for three or four years at that point.

It was already established, but companies still hadn’t fully caught up and created their own in-house teams. We were really at a good point, and got really lucky to ride the wave, honestly. That’s probably why the company survived at all.

If I was trying to do the same thing today, starting from scratch with no knowledge of the industry, it would be a lot harder to replicate that; at this point, a lot of the companies that used to hire agencies like Magnetic Bear Studios, or small agencies in general, now tend to have their own in-house group of developers and designers who can lead their mobile effort.

It’s not longer a fad, it’s no longer hype, it’s just like the web was five years ago. It’s expected that if you have some sort of electronic presence that you definitely need a mobile strategy of some sorts. The landscape has certainly shifted.

Consultancy Challenges (10:27)

Q: Right, that makes absolute sense. What was your key takeaway in terms of challenges of running an iOS consultancy during those three and a half years?

JP: Trying to put myself back in those shoes from a few years ago, which is becoming increasingly harder to do, I recall some of the challenges or at least some of the things that I used to complain about, at least internally. Even in 2011 and 2012, I thought I was too late to ride the hype and wave for mobile.

There were a handful of agencies that had popped up around the world, and I was finding myself competing with the more established smaller sized agencies that had done what I was doing. That was pretty tough, and I can only imagine someone trying to do the same thing today would have a much harder job of actually trying to pull that off. In retrospect, although I was doing that not at the very earliest time and maybe not at the perfect opportunistic time, the conditions were still really quite good to pull that off.

You look at the landscape now and not just in agencies but just in mobile development in general and the race to the bottom is really won. Very few people can make a living off of apps on the App Store and the strategies haves completely shifted towards either subscription based or in app purchases. Ultimately, we were already starting to see that trend three or four years ago, and that’s one of the reasons why I essentially grew a company as a contractor and as an agency rather than as a company developing apps directly for the App Store.

In a sense, a lot of these companies that were hiring us knew they needed a mobile presence, but didn’t know exactly how to monetize it and probably wouldn’t have been able to in a very good way anyway. Really the only way to bootstrap a company from the ground up was to get people who were willing to invest because end users of applications weren’t necessarily ready even at that point.

Starting at Realm (13:09)

Q: Tell us how you ended up at Realm and tell us a little bit about your role there.

JP: In 2013 and 2014, we were starting to see the limits of Magnetic Bear Studios and growing that as an agency. I was really lucky to have some very talented people, but ultimately, all the people in my team still had somewhat of an entrepreneurial spirit and building apps as an agency tends to wear you down. It’s not exactly for everyone, even though I do look back fondly on those days where we could have a wide variety of projects on an ongoing basis. It really kept you on your toes and kept you interested.

Ultimately, towards the end like I mentioned, I was doing more of the business side of things and I A) wasn’t very good at it and B) I didn’t enjoy it. I was looking to get more into a technical role and that’s where I saw that this stealth company in Silicon Valley was hiring. If you know me at all, especially in those days, those kinds of words were really a big turn off: “VC funded stealth company.” Back then, I was really all about bootstrapping a company, creating value right from day one rather than the promise of value for an investor.

Ultimately, the closer I looked at this company who didn’t even broadcast their name at that point, the more I appreciated that different types of companies and products require different types of business models and even bootstrapping processes. You look at a database that needs to be built from bottom up, and your minimum viable product there is a fully ACID compliant database. You can’t really just build half a database and release it out to the world.

If you’re going to disrupt a technological space like that that needs several years and a small team at least of upfront investment before you can actually start to monetize a product out of it, then venture capital or investment based strategies is really the only way unless you take more of a side project approach to building something like that. To be honest, I don’t really think that that would have worked with such a core fundamental technology as a database. I really appreciated and respected the approach that Realm took at the time.

I was chatting with the founders and at that point they had been building Realm’s underlying storage engine for the past three years. Starting summer 2011 and going through Y Combinator, that’s where they were really starting out. They had this concept of modernizing database engines, especially for embedded databases, in a way that really didn’t make sense 15 years ago when SQLite was developed. If anyone’s been in the mobile development industry here, even in the embedded space in the last decade and a half, they know that SQLite is a very robust type of software, but it didn’t always start off that way.

It needed to grow into that level of stability. Also, it was designed at a time when we didn’t have the smartphone revolution as we know it today. We didn’t exactly have iPhones or Android phones or even this concept of model oriented or object oriented, object relational mapping libraries. You look at Core Data, I mean you had web objects and stuff, but not so much in the embedded space.

You look at Core Data coming along and the genesis there really had the right ideas, where it was like, “Let’s build something that’s object oriented because that’s how we want people to write software.” Not this kind of table and row, kind of snapshot based system where you essentially query some external thing to ask what its state is and then you go and translate that. If you actually had a system like an ORM that can keep that in sync with the models and the objects that are really integral to object oriented programming, then you’re really burring the lines and at that point persistence and more database type concepts, like ACID compliance, then becomes more of an implementation detail.

You look at the progression of technology in the 90s, the early 2000s, even the late 2000s and people really built amazing things on top of the fundamental technologies that they had. Then you get to 2010, 2011 and you really reach a tipping point where you’re starting to stretch the useful space in which those technologies designed ten years ago really start to apply and some of these concepts start to break down. You start to have arm based chips that are really prolific. You have SSD’s, you have vectorized operations and CPU’s, and these are all things that if you were designing a low level database today, you would really want to take advantage of those things.

It’s hard to take this older system that’s been battle tested, made very robust over the course of a long set of years, five, ten years, and then say, “Okay, well let’s re-architect this.” No matter what, at that point, anything you touch has an impact on the memory layout, has an impact on small things. That when you consider the scope of how wide it’s distribution is, you really start to realize that it’ll have very strong knockout effects. It’s very likely that it’ll break something somewhere.

You’re put in this impossible situation where you can’t innovate without breaking things, but people rely on these foundational technologies to be robust before anything else. Robustness over feature completeness, really. That’s where throwing out that concept, basically starting from scratch, really comes in handy. Where you can consider some of the research that was done in modern times about multi-version concurrency control, about copy on write semantics, about making the most of SSD’s and modern components, and modern computing architecture and really build something from scratch.

That’s what Realm did and that’s what ultimately drew me over to them. This promise of starting fresh. If we were to build something, a database for the smartphone revolution era. That’s what really got me in.

Working at Realm today (20:41)

Q: What’s your role at Realm today?

JP: My role today is leading the Apple software teams. We have two core products and these are user facing products. Those are Realm Objective-C and Realm Swift. I lead the team that’s responsible for building those, but ultimately the kinds of work that we do, that our team does, which is about seven or eight people at this point, spans everything from writing cross platform low level functionality, even at the Core Database level that can be used for other products, like Realm Java or Reactnative.net.

It spans from that to doing specifically Objective-C and Swift stuff, or even the Realm browser which is a Mac app that we have, to doing documentation tests, sample apps, tutorials, and community outreach as well. We don’t have any pure Realm evangelists in the company. We just have the engineers that are working on the product.

Why work at Realm? (22:03)

Q: Give us the elevator pitch for Realm. If I’m an iOS developer and I’m looking at my options in terms of data store, why should I pick Realm?

JP: The main thing that we strive for there is simplicity. Simplicity on so many levels. You look at a system, like a traditional ORM, and there’s so much intermediate state that needs to be maintained to keep the object side and the relational side in sync. That’s where you start to introduce faults, where you’re accessing something on one thread and another thread was mutating it, but because of the shared global mutable state that’s in common there, halfway through an operation you hit a fault and say, “Well, another thread deleted an object in a completely different context and I’m suffering the consequences there.”

Simplicity at that level. Everything from the most foundational layers up. Where if you lose this object relational mapping it’s just objects the whole way day. That simplicity translates into an easier way to reason about what’s happening under the hood. It translates to performance as well because you’re doing less work. It translates to safety because when the very lowest levels are built with that concurrency in mind, then you don’t need to have multiple layers where you need to ensure that that safety is preserved if it’s kept as simple as possible.

There’s simplicity on that end of the foundational technologies, and there’s also simplicity in terms of usage and the API. One of the things that we really strive for, more than feature completeness or even performance, is ease of use and maintainability. Initially when we released Realm Objective-C, which was our first public product, we had one idea of what the developer community was looking for and then they responded with something, not exactly completely different, but it did surprise us.

Initially we thought that speed was a very important factor and after talking to a lot of developers, a lot of them, if it’s fast enough, that’s all that matters. As long as they can have their UI table view that swirls at 60 frames a second, they’re happy. It doesn’t really matter if you can perform this complex object graph type query where you’re traversing many object graphs in a fraction of the time that it would take you with an ORM, that doesn’t matter as much for the vast majority of developers.

Another thing that we found is that a lot of the ping points like performance that people tend to experience with traditional solutions, people tend to internalize that. Where it’s part of normal development. You don’t exactly remember exactly how much effort it was to get things working within a reasonable time frame. We found out that performance as such wasn’t an end goal for most people, that it was actually more simplicity, maintainability, and the ability to have code that lasts for a long time, that doesn’t need to be constantly updated.

Core Data is complicated (25:41)

Q: That’s very interesting. A complaint of many new iOS devs is, “Why is Core Data so complicated?” Is it fair to say that Realm is competing with Core Data?

JP: Absolutely. It depends from what angle you look at it. Realm is definitely an object data layer and you can consider Core Data to be very similar in that sense. The types of things that you would do with Core Data, Realm is really a suitable replacement. There are design decisions that we’ve done to make sure that people migrating from Core Data to Realm have an easy path there. I’d say, especially in recent weeks, ever since Realm launched the Realm Mobile Platform, Core Data starts to really be more of kind of a subset in competition to Realm rather than an exact kind of competitor, if you will.

The Realm Mobile Platform, you can look at it as sync. It’s data synchronization between devices, between servers, but really if you peal back the layers, it’s a lot more than that. You look at apps today and a lot of the complexity goes into things like parsing JSON, doing network requests, doing caching, knowing when to pull, when to push, when to cache, when to invalidate a cache, what parts of that cache to invalidate, et cetera.

You look at all of that cruft and if there was a way to cut down on a lot of that complexity, app developers would really be able to focus a lot more on the value that they want to add in their app, on the kinds of functionality that really sets them apartment from the competition, compared to doing all this boiler plate stuff that has such a high investment cost and also so many ways in which things can go wrong.

You consider a traditional application that uses rest API’s and JSON serialization, caching layer, and you look at a round trip between keeping data that’s on the server, which is typically the canonical source of truth, in sync with at least what the application presents the user. There’s so much cruft that needs to go into that there if there was a way to cut down on that, you would really drastically simplify the way that people build apps today.

In a sense, Realm isn’t exactly competing with any one thing, we’re really trying to upset the status quo so that you can drastically simplify your apps, make them a lot more robust, while adding things like real time functionality with very minimal effort. That’s really kind of the approach that we’re taking.

Realm Performance Relative to Core Data (28:42)

Q: While performance is not your number one goal, simplicity is, how does Realm perform relative to Core Data? Are there any areas where Core Data outperforms it?

JP: There certainly are. There are fewer and fewer. Our mantra is to be within the same ball park as SQLite, for the most part. When it comes to benchmarks, one of the reasons why we’ve stopped producing benchmarks, at least externally, is because you can always make them look good for you, or for your use case. If you set the right size, the right transaction, or the right amount of concurrency competition that’s happening, you can make one product look much better than the other. We can tweak those numbers to make any product clip better than the rest of its competition.

We really do take performance very seriously and we have a number of performance tests that we will at least make sure that we don’t regress on a lot of key metrics or key operations. I’d really leave that to the end user to determine for themselves if for their use case Core Data, SQLite, or Realm is much faster than anything else. What I will say is that we really do strive to hit a sweet spot where we can be as performant, if not more than SQLite or Core Data, for the vast majority of types of architectures that mobile developers tend to use these days.

But I’d stop short from saying that any one product is considerably faster than anything else. You even look at Core Data and some of the things that it does is actually faster than Ross SQLite, depending on the type of operation you’re doing. I’d really leave that up to end users to determine for themselves.

Back end Realm usage (30:46)

Q: Realm is on several different platforms. Is Realm ever used as a web back end as well?

JP: It certainly is in several ways. One is in the same way as MySQL, or PostgreSQL, even SQLite can be used in server circumstances, where you need a transactional ACID database. Realm fits the bill there. There’s another angle to this, and this is probably the more common one, and that’s with the Realm Mobile Platform. Having a database on the server side that can communicate and coordinate with remote clients to facilitate things like synchronization and even things like event handling, that’s a much stronger use case.

That being said, Realm on the server is still in its early days, at least from a product standpoint. I wouldn’t say from a technological standpoint, but from a product standpoint, for sure. The only official language SDK that we provide really for the service is JS, is JavaScript. You look at the rest of our product line and you’ve got C sharp for .NET, you’re got Java for Android, you have Object C and Swift for Apple platforms, so that’s iOS, Mac OS, TV OS and Watch OS. None of those are designed to run say in a Linux infrastructure. The one product that we’ve specifically honed for that is the no JS SDK.

Q: The reason I asked the question is because Realm started as an embedded database for mobile devices. It seems like over time the database is spreading its wings and becoming a more general purpose database, is that a fair characterization?

JP: You could say that. Another way to put it is that Realm really started with the fundamental building blocks of building a very robust distributed database. That’s where the whole transaction log comes into play where we preserve the semantic intent of all the operations. This comes in nicely when you have distributed systems, especially when you need really high partitioning tolerance where devices can be offline at any time and you don’t want the performance of the database to take a hit for that.

If you’re in an intermittent connectivity area, which is basically all of mobile devices, all of mobile networks, because even as you’re walking through a building, even as you’re walking down the street switching between cell towers, you have intermittent connection, the last thing that you want is for your on device performance to suffer. Since the very beginning, since the genesis of the project, we’ve built everything in preparation for having a really robust distributed system. I’m really excited that we can finally offer that as a product now.

Realm Mobile Platform (34:13)

Q: Tell us more about that product. The big announcement from Realm in the last couple months was the Realm Mobile Platform. What is the Realm Mobile Platform?

JP: The Realm Mobile Platform is essentially exactly what you said. It’s taking Realm on device and making it spread its wings. The mobile platform lets you do things like device-to-device synchronization, it lets you do things like real time collaboration, data back up and collaboration.

If you wanted to build something similar to Google Docs where you have multiple users that have different permission levels that are concurrently accessing a document making changes and having it all resolve automatically, those are really some of the main use cases that we’ve honed the Realm Mobile Platform to support.

There’s a number of other things that you can do. Earlier I was saying if you wanted to cut down on the complexity of your local stack, you remove the necessity to do JSON serialization, network requests, cache logic for invalidation and cache checking, if you wanted to do all of that, you could move a lot of the functionality from your app into the server and then at that point all the server really needs to do is to write objects to its local realm. Because of the way the Realm Mobile Platform works, that data is automatically available on the devices that have access to that realm.

Imagine you had an app that took a picture of a wine label and automatically gave you information about that wine bottle. What you could do is you could write a local object in your local embedded database, local embedded realm on your device that has the picture for that. Because of the way sync works, the server would then very quickly have access to that information, it could listen to changes that are happening, see, “Oh, this device just added a picture. Let me do all of the back end processing that I need to do, talk to several different servers, some of them maybe online, offline, some of them may have high latency,” whatever.

For the most part because it’s doing this work on a back end infrastructure, it typically has a much better internet connection so it can do a lot of this work and then once the response comes back from whatever this server that was listening to this change did, it can just write the response to its local realm object and then the device would then see that change locally in its local data store. You can really cut down on a lot of the complexity by sharing, say you have a cross platform app, you can share all of this business logic on the server where you don’t need to duplicate it and rewrite it in Java or Swift, or whatever, while still maintaining all of the functionality.

Q: Is it fair to say that the Realm Mobile Platform is in the same category as something like CloudKit, or what Parse was, or Firebase?

JP: There’s a lot of overlap when you mention products like those. There’s a lot of really key differences. You’re looking at CloudKit and for the bulk of what it offers, it really is for the most part a key value store. You have things like iCloud document syncing, but that’s really done more of on a very course grain document level. Then you look at something more like Firebase and it doesn’t really have much in terms of persistence or database type operations. You still need a solution for that locally on device.

Then you get into other differences, such as the merge algorithm for one, how do conflicting changes get resolved? This is something that Core Data with Core Data Sync tried to do with its ubiquity store. Ultimately there were a few architectural challenges with getting that to work whereby building that at an ORM level rather than at a database level, you lose the transaction log of the database. You lose all of the fine grain operations that are done.

Whereas with Realm, if you actually build synchronization and merge conflict resolution from the very lowest levels up, you maintain that intent, that semantic intent of operation. When you have things like a move or a re-name or sub-string operations, if you were doing that at a high level without access to a transaction log, you have no idea if a change happened because of a move, because of a modification.

You can apply heuristics and try to guess what actually happened and what the intent of the user was, but as soon as you have a moderately complex use case, and you really don’t need to look very far to get that, that system starts to break down. That’s where you get things like incorrect merges. That’s really why Realm went and essentially reinvented the concept of an end-to-end database was to provide the fundamental tools to be able to do this efficiently and easily.

Q: Would it be fair to say that Realm is more robust than those other two solutions because it’s an end-to-end solution?

JP: I would certainly say so, but like you said, that’s generalizing certainly. Depending on your use case, you may not necessarily think so. I’m hard pressed to think of anything specific right now, but the last thing that I want to say is that this hammer that we’ve built, which is Realm, is a tool for all problems out there. The old adage, “When you have a hammer, everything looks like a nail.”

Sure you can do a lot with Realm. One of the reasons why we focused on a fundamental low level side of things was that it’s easier to build flexible solutions on top of robust primitives, rather than say have a really easy to use but super high level wrapper that’s a black box and that we don’t expose anything below the very surface. That’s what I say.

Q: Can developers install the Realm Mobile Platform back end server on their own servers or do they need to go through Realm?

JP: No, they actually have to install it on their own servers. We don’t offer a hosted solution right now. This is both a good thing and a bad thing because if you’re looking to own your data, if you’re looking to have full flexibility in terms of how it’s deployed, how it’s exposed, how it’s maintained, then the way that we currently offer the Realm Mobile Platform is really ideal for you.

This tends to be what most larges businesses tend to look for. That’s really who we’ve aimed to target from the very beginning. If you’re looking for a hosted solutions, we don’t offer anything right now and we are keeping our finger on the pulse in terms of how enterprises are using us in their own deployments to be able to figure out, “Okay, well is there a way that we can learn from this to maybe offer a fully hosted solution?” If that comes, it’ll come later.

Q: It sounds like you’re doing it in the right order. A lot of people felt very burned by what happened with Parse and that lead to a lot of consternation about having the data store itself be on someone else’s hardware. It sounds like you’re going about this in the right direction.

JP: We think so too, but obviously that’s more of a matter of opinion. If you’re a single person developer, managing your own infrastructure on top of your app might not necessarily be the answer that you’re looking for. We completely understand that, but on the other hand that’s exactly why we chose to fully open source the local embedded database. If you’re running Realm on your device, if you’re not using Sync, you’re using an entirely open source solution.

At that point there are no shackles, there is no lock in. You can do absolutely whatever you like. You can fork it and modify it to your heart’s content. We really hope that individual developers aren’t taken aback by our lack of hosted solution offering. On the other hand, it also helps us really focus on delivering the absolute best experience for the people are willing to run this on their own infrastructure for now. We can then apply those lessons maybe later on when we’re ready to offer maybe a different type of product. The future will tell.

Q: Okay, so a bit of a cliff hanger. Maybe there will be a hosted solution in the future but that’s not yet determined, is that what we’re hearing?

JP: We’re not offering anything right now and we are focusing on self-hosted solution at this point.

Realm and the Open Source Community (43:57)

Q: Realm, the main database, is open source. Realm is very involved in the open source community. Tell us a little bit about that.

JP: One of the reasons why I wanted to join Realm in the first is really this ethos of contributing and being a participating member of the community. You’ll see a lot of companies that are in the developer tools space who might sponsor a lot of conferences, or they might have booths at trade shows and things like that. Although we do that from time to time, we aim to be a contributing member of the community. Not only does that help us to better keep in touch with our user base, but it also helps users because they can interact with us in a more fluid way.

That’s why we’ve been running the Swift Language User Group and hosting those videos on our site. Whenever possible with conferences, we’ll provide video recording, hosting, and transcription services for free, rather than just asking them to put our logo on a banner and giving them money for it. We want to be an active participant. That’s more on the community engagement side.

On the open source side, the high level SDKs have been open sourced since day one. Realm Objective-C, Realm Swift, Realm Java, JavaScript, .NET; all of that’s been open sourced. The core engine, however, was closed source for the first two years. There’s a handful of reasons there, but ultimately we want to be very transparent in terms of what our business model is and where Realm fits in, not just as a community member, but as a company.

You mentioned some other companies earlier and how people got burned. If you’re not fully clear about what your business model is, even though some people might not exactly be fully aligned with those priorities, then you’re really not being transparent. That really hurts people who are trying to build a business of off our products and our offerings. In that sense, keeping the core closed sources was an important message there, since we certainly did need to clean it up before it was available to the public, but also that’s where our intellectual property was. Being transparent about that was way more important than anything else.

Q: That makes complete sense. I want to also talk a little bit about some of your open source projects, like PeerKit and SourceKit. Tell us a little bit about them.

JP: PeerKit and SourceKit are really personal projects of mine. There are a handful of non-database related company projects that we have open sourced, like Jazzy, SwiftLint, and the Realm Browser.

Jazzy is an Objective-C and Swift documentation generator. That really was birthed out of a necessity for the company to have documentation for our own product.

Two years ago, at WWDC 2014, Swift was announced. This was funny timing because this was right around the time that we were gearing up to release Realm Objective-C. We said, “You know what, let’s just wait a week and see what happens at WWDC.”

Sure enough, Apple announced this new language and blindsided everyone. We essentially gave ourselves a week to support the language, and then released our product. Building a product that’s birthed into this new language, you need documentation for that language. That’s where Jazzy came about’ it was for our own business need to document our product, and it was a really fun project.

I loved hacking on that, reverse engineering Xcode, reverse engineering SourceKit, back when it was closed source and trying to figure out how to build a robust documentation generator. Another option would have been to build a parser for Swift, but if you look at Swift’s tumultuous history in the last two years, it would not have been fun at all.

Swift is a much more complex language than Objective-C, where in Objective-C you have very simple syntax. I’m not saying that it’s easier to learn at all, but from a parsing standpoint it’s a lot easier to par Objective-C than Swift. For one, you don’t have things like struts and language, sure there’s C++ struts and things like that. Another point is that you only ever have one level of nesting in Objective-C. You have classes and they can have properties, but you can’t have a class within class within property, or a closure within a function within a class.

Whereas with Swift, you have so much flexibility and so much complexity that follows that building a parser would have been a nightmare. The investment of reverse engineering Xcode and SourceKit seemed like a smarter choice and in retrospect I think it was.

There’s Jazzy and then there’s SwiftLint which is more of a linter for Swift that was really done more out of my own frustration that people hadn’t built one yet. I built SwiftLint in a weekend in May 2015 because I was giving a talk on fixing the Swift tooling problem at UIKonf in 2015. My point at that talk was really to say, “Sure, Swift is a new language and therefore has a lot of room for improvement, especially around tooling,” and also that, “We shouldn’t necessarily have to rely on Apple to build that tooling.”

I’ve been saying for a while that using something like SourceKit that you could pretty easily built a linter because Apple has gone through the efforts of exposing a way for people to build tooling for Swift. I’d been saying that for over a year at that point saying, “Hey, someone should really just built a linter on top of SourceKit.” As I was writing the talk, as anyone who’s given a talk before, you really want to procrastinate and do anything but. I ended up writing that on the plan just to show, to kind of make my point from the talk that, “Hey, we don’t have to wait for Apple to build tooling, we can do it ourselves.”

I built it then and it really exploded from there, mostly because no one else had really done it. Since then, there’s some terrific formatters and linters that have come out, but I think the community really latched on to SwiftLint and now it’s pretty actively used project.

Conclusion (52:04)

Q: Jazzy and SwiftLint are really fantastic projects and I recommend everyone listing check them out. Is there anything about Realm that we didn’t cover that you really want developers to know about it?

JP: That’s a good question. I think we really touched on a lot, especially around the Mobile Platform. This is really on us at Realm to do a better job of talking about this, but there are a handful of limitations in Realm that albeit annoying, are actually there for a very good reason. It’s not to say that they can’t be lifted, eventually, just that there’s a reason why they’re there.

One thing that comes to mind is thread confinement. Earlier you may have heard me say that Realm was a threadsafe database. What I mean by that is that transactions are isolated and any changes that are being done by one thread, or a different process, or even a different device that’s being synced, won’t effect your current consistent view of the database. We do this via a number of different ways: multi-version concurrency control, snapshots and copy-on-write.

One of the side effects of this is because Realm is an object graph database and that one thread may be accessing a different snapshot of the database than another, passing an object across a thread is actually very dangerous. What could happen is that as you’re passing this object, for a person that has a child property, a relationship or a link, what happens when the two different snapshots between those two threads have different values for that relationship? Or even different values between scalar primitive properties, like a String or an Integer?

You get into a situation where you don’t want to have a later snapshot try to access an object from an earlier snapshot. You get into this dependency resolution state in which it’s impossible to have a general solution. That’s why for safety we say, “Every object is confined to its own thread.” That’s not necessarily done out of our inability to let you access the object from another thread. It’s done so that people don’t accidentally write unsafe code that expects thing to be a certain way and then there data just changes from under them.

That being said, there are ways that we can improve ways to work around this, but you’re never going to be able to really pass data between threads or freely write data outside of a write transaction and that’s do to wanting transactional mutations. So if you need to change two things that if your app crashes half way through, then you don’t end up in this inconsistent state.

There’s a bunch of these limitations that people love to complain about and they are annoying (I’m the first to admit that), but the solution isn’t as simple as giving people the first solution that would come to mind. We’re working on solutions for a lot of these things and I really just appreciate everyone’s patience in understanding the complexity of the issues.

Q: I really recommend everyone check out Realm. How can our listeners get in touch with you, and where can they follow you?

JP: People can get in touch with Realm via Twitter, @realm or Realm.io on the interwebs. Email us [email protected] if you have any questions or issues. For me personally, I’m @SimJP on Twitter and then “JPSim” on everything else, like GitHub.

Q: JP, it’s been fantastic having you on the show. Thank you so much for taking the time, and I hope that all of our listeners learned a lot about Realm.

JP: Dave, thank you for having me, and I hope this was helpful!

Read more

Realm Objective-C & Swift 2.2: Objects across threads, sort over relationships & more!

by /

One of Realm’s earliest design goals was to have a consistent and straightforward threading model. Today, we’re releasing version 2.2 of Realm Objective‑C and Realm Swift to improve our threading model by introducing a mechanism to safely pass objects across threads. We’re also releasing the ability to sort by relationship properties, some sync improvements and a few bug fixes. Read on to learn more!

Following the Thread

Today, Realm makes it easy and safe to work with your objects in multiple threads. Our current multi-threading support is the result of years of steady effort and deliberate design decisions (because threading is hard).

Our previous Threading Deep Dive article lays out many of the philosophies behind the way Realm safely handles concurrency without requiring users to think about locks or resource coordination, all while providing a consistent view of the entire Realm and its object graph. Notably, this design architects away the concept of ‘faults’ present in other ORMs and data frameworks 💥.

Our documentation on Threading is also a must-read resource to truly understand how to best use Realm in concurrent environments. With these resources, you can make extremely productive use of multi-threading and Realm. But until now, the objects you use haven’t been able to cross threads.

Thread Confinement

If Realm’s so thread-safe, why do I get exceptions when I try to pass my object between threads?!

In order to provide the consistency and safety Realm guarantees, a simple constraint is imposed: all Realms, objects, results, and lists are confined to the thread on which they created.

It’s important to understand that thread-confinement isn’t a temporary limitation based on Realm’s internals, or an artificial constraint, but rather a crucial part of the design that makes it easier to write correct code.

In fact, freely passing Realm objects across threads would be very easy for us to implement, but would have tradeoffs that would render it very dangerous and unpredictable to use.

Realm is a transactional database (think the scope of a write transaction) that allows an app to crash halfway through a write without having partially valid data written to disk.

To have isolation (the ‘I’ in ACID), changes in different transactions aren’t reflected in other transactions until those transactions advance to the latest version. Otherwise, you’d get ‘faults’, which Realm doesn’t have thanks to this design.

To aid in this isolation, Realms on a given thread are only ever at the same version (under the hood, there’s only a single Realm per-thread). Passing a Realm, or Realm objects/queries/etc freely across threads would mean that you’re mixing different versions of the database together, which would lead to very surprising results. For example, passing an object to a thread that has since deleted it could cause a crash, or it could change values as it crosses the thread boundary, or an object graph could have different relationships in either transaction version.

The Old Way to Pass Data Across Threads

Up until now, in order to pass data across threads, you’d need to pass data that wasn’t backed by a Realm.

This was commonly done by passing unmanaged instances of Realm objects, or reading data out of Realm-backed properties such as a primary key value.

let realm = try! Realm()
let person = Person(name: "Jane", primaryKey: 123)
let pk = person.primaryKey
try! realm.write {
  realm.add(person)
}
DispatchQueue(label: "com.example.myApp.bg").async {
  let realm = try! Realm()
  guard let person = realm.object(ofType: Person.self,
                                  forPrimaryKey: pk) else {
    return // person was deleted
  }
  try! realm.write {
    person.name = "Jane Doe"
  }
}

However, this doesn’t work at all when objects don’t have a primary key, and may result in you operating on stale data. Passing something other than a Realm Object, such as a List, Results or LinkingObjects, also can’t easily be done with this approach.

Using Thread-Safe References

Now, you can create thread-safe references for all types that were previously thread-confined within a Realm, and passing objects between threads becomes a simple, three-step process:

  1. Initialize a ThreadSafeReference with the thread-confined object.
  2. Pass that ThreadSafeReference to a destination thread or queue.
  3. Resolve this reference on the target Realm by calling Realm.resolve(_:). Use the returned object as you normally would.

For example:

let realm = try! Realm()
let person = Person(name: "Jane") // no primary key required
try! realm.write {
  realm.add(person)
}
let personRef = ThreadSafeReference(to: person)
DispatchQueue(label: "com.example.myApp.bg").async {
  let realm = try! Realm()
  guard let person = realm.resolve(personRef) else {
    return // person was deleted
  }
  try! realm.write {
    person.name = "Jane Doe"
  }
}

Real World Example 🌏

Thanks to RealmTasks being open-source, the pull requests replacing existing thread passing code with thread-safe references can be viewed on GitHub: iOS PR #374.

Here’s the relevant code, where we automatically deduplicate a List property, performing the write transaction on a background thread:

realm.addNotificationBlock { _, realm in
  let items = realm.objects(TaskListList.self).first!.items
  guard items.count > 1 && !realm.isInWriteTransaction else { return }
  let itemsReference = ThreadSafeReference(to: items)
  DispatchQueue(label: "io.realm.RealmTasks.bg").async {
    let realm = try! Realm()
    guard let items = realm.resolve(itemsReference), items.count > 1 else {
      return
    }
    realm.beginWrite()
    let listReferenceIDs = NSCountedSet(array: items.map { $0.id })
    for id in listReferenceIDs where listReferenceIDs.count(for: id) > 1 {
      let id = id as! String
      let indexesToRemove = items.enumerated().flatMap { index, element in
        return element.id == id ? index : nil
      }
      indexesToRemove.dropFirst().reversed().forEach(items.remove(objectAtIndex:))
    }
    try! realm.commitWrite()
  }
}

Sorting By Relationship Properties

Until now, Realm Collections could only be sorted by “direct” properties.

Starting with Realm 2.2, you can now sort collections by properties on the other side of an object’s to-one relationships.

For example, to sort a collection of Persons based on their dog property’s age value, you could call dogOwners.sorted(byKeyPath: "dog.age"):

class Person: Object {
  dynamic var name = ""
  dynamic var dog: Dog?
}
class Dog: Object {
  dynamic var name = ""
  dynamic var age = 0
}

realm.beginWrite()

let lucy = realm.create(Dog.self, value: ["Lucy", 7])
let freyja = realm.create(Dog.self, value: ["Freyja", 6])
let ziggy = realm.create(Dog.self, value: ["Ziggy", 9])

let mark = realm.create(Person.self, value: ["Mark", freyja])
let diane = realm.create(Person.self, value: ["Diane", lucy])
let hannah = realm.create(Person.self, value: ["Hannah"])
let don = realm.create(Person.self, value: ["Don", ziggy])
let diane_sr = realm.create(Person.self, value: ["Diane Sr", ziggy])

let dogOwners = realm.objects(Person.self)
print(dogOwners.sorted(byKeyPath: "dog.age").map({ $0.name }))
// Prints: ["Mark", "Diane", "Don", "Diane Sr", "Hannah"]

To accomplish this with previous Realm versions, it was necessary to store the dog.age property on the Person object, or to sort outside Realm, losing all advantages of Results.

This is an API breaking change: instead of referring to ‘properties’, we’ll use the more general ‘key paths’ term.

  • The following Objective-C APIs have been deprecated in favor of newer or preferred versions:
Deprecated API New API
-[RLMArray sortedResultsUsingProperty:] -[RLMArray sortedResultsUsingKeyPath:]
-[RLMCollection sortedResultsUsingProperty:] -[RLMCollection sortedResultsUsingKeyPath:]
-[RLMResults sortedResultsUsingProperty:] -[RLMResults sortedResultsUsingKeyPath:]
+[RLMSortDescriptor sortDescriptorWithProperty:​ascending] +[RLMSortDescriptor sortDescriptorWithKeyPath:​ascending:]
RLMSortDescriptor​.property RLMSortDescriptor​.keyPath
  • The following Swift APIs have been deprecated in favor of newer or preferred versions:
Deprecated API New API
LinkingObjects​.sorted(byProperty:​ascending:) LinkingObjects​.sorted(byKeyPath:​ascending:)
List.sorted(byProperty:​ascending:) List.sorted(byKeyPath:​ascending:)
RealmCollection.sorted(byProperty:​ascending:) RealmCollection.sorted(byKeyPath:​ascending:)
Results.sorted(byProperty:​ascending:) Results.sorted(byKeyPath:​ascending:)
SortDescriptor(property:​ascending:) SortDescriptor(keyPath:​ascending:)
SortDescriptor​.property SortDescriptor​.keyPath

Realm’s commitment to SemVer means that these deprecated methods will continue to be supported until Realm hits 3.x.

Other Changes

Sync Breaking Changes (In Beta)

  • Underlying sync engine upgraded to version BETA-6.5.
  • Sync-related error reporting behavior has been changed. Errors not related to a particular user or session are only reported if they are classed as ‘fatal’ by the underlying sync engine.

Bug Fixes

  • Setting deleteRealmIfMigrationNeeded now also deletes the Realm if a file format migration is required, such as when moving from a file last accessed with Realm 0.x to 1.x, or 1.x to 2.x.
  • Fix queries containing nested SUBQUERY expressions.
  • Fix spurious incorrect thread exceptions when a thread id happens to be reused while an RLMRealm instance from the old thread still exists.

Legacy Swift Version Support

We’d like to remind you that we will continue to support Xcode 7.3.1 and Swift 2.x as long as we can, but encourage all our users to migrate to Swift 3 as soon as possible.


Thanks for reading. Now go forth and build amazing apps with Realm! As always, we’re around on Stack Overflow, GitHub, or Twitter.

Read more

Introducing the Realm Mobile Platform

by /

 

Introduction

If you’ve been following Realm for a while, you probably had an idea that the Realm Mobile Platform was coming. It took some pretty hardcore research getting it to actually work, as it’s a project that has been underway for at least two years.

Realm’s goal has always been to make it possible for people to build better apps: ones that have amazing live and interactive features, that are more easily and quickly buildable, since those kind of apps are currently far too hard to build.

Realm Mobile Database

Realm has quite a long history, and today we’re doing great! We have offices in San Francisco and Copenhagen (some of us have a Danish background). We have around 50 employees now, and we’ve raised money for ourselves. But it didn’t start off like that! It’s funny that it all started with just me and my co-founder, who had this crazy idea while sitting in Denmark that we wanted to build a mobile database.

We applied to Y Combinator. I know there are people who say that if you apply to Y Combinator, you need to know all the right people and fit a certain profile, etc, but we were two people sitting totally alone in Denmark. We didn’t know anybody, and we had no credentials other than our skills and the fact that we’d worked at Nokia doing mobile stuff for a long time. We were total nobodies, and very naive.

Still, we sent our application video, and it was amazing that it was well received! It’s a long story from there, but we tried harder and got everything up and running. We got started and created the mobile database. It was the core engine of what we wanted to do.

Being young, when we started we thought, “How long time can it possibly take?” We both had lots of experience in this field, so we imagined that it would take around half a year before we would have something we could sell. Here we are, several years later, and we actually have a decent product! We launched in 2014, almost three years after starting the company. It takes a long time to build an embedded database, especially if it’s a good one. It’s probably a good thing we didn’t know that when we started, because we may not have started at all. Now that we’re here, it’s really cool!

Now we have this technology in place, developed, fully open source, and it’s pretty awesome. We promised from the beginning that we would open source it, and now you can do whatever you want with it. It’s always a pleasure to give things to the world.

What is most satisfying for me is how much people love it. We get tons of tweets and feedback where people talk about their experience with how cool Realm is. We’re also used pretty much everywhere, which is amazing. We’ve changed so many apps!

The Realm Mobile Platform

After releasing Realm Mobile Database, we needed a next step. It’s great to make a cool database and give it to everybody for free, but that’s not how you create a sustainable company. We needed the Realm Mobile Platform.

One anecdote that most people won’t know is that the original inspiration for Realm came because I was building a text editor at my own company. It was a collaborative text editor with git built in (before git was invented), and I wanted to have real-time collaboration. No database was fast enough to have the type of real-time collaboration I was looking for, so I wanted to make it for myself. I started building that, but the whole idea was to build an ideal database that allows people to share data in real-time. It turns out that it takes a long time, because you actually have to build a database first!

We got a lot of requests for synchronization, and a lot of people wanted to know when it was coming. Today, you want real-time stuff that just works, where you can update and have everything be reactive. It’s currently difficult to make things real-time in your app; you should be able to just connect your UI directly to the data and it should just update. With our platform, you can have these features with hardly any code at all, and it’s easy to do.

You want a reactive setup, where you can just create all your objects, connect them to your UI, and it should work. From my original vision, it should be collaborative. Everybody is getting mobile phones, and we want people to work together. Every app should be collaborative. It should help people to create, so we’re not just sharing cat pictures, but we should have quality apps that actually help people do things. We want everybody to have a seamless experience that just works, online and offline. That was the vision, and it’s what we’ve heard from users.

Realm Mobile Platform is the counterpart to the Database, because you can only do offline for real if you have a real database on the device. That’s key, otherwise it always falls through and the experience is not good. The magic of this is that you can have a database, but the database is just a webpage. Instead of opening the database with a filepath on your disc, you just open it with the URL. You open the shared Realm, and you can open one, ten, or hundreds of them. Anybody who opens the same URL will have the same data, at the same time, in real-time. It’s a very simple concept, but it’s pretty amazing when you actually work with it.

The idea is that the database is offline-first by nature, because it’s real object database. It’s not a SQL database with OM layers, copying data back and forth with inconsistent data. It’s a real database that’s relational, so you can have links to everything. You also have chain notifications that allow you to do reactive things. Now you can totally abstract away the network handling. Instead of thinking about networking, you simply open the Realm at the URL, and it’ll naturally synchronize with whoever else just opened the same Realm.

Other cool things include encryption, which you definitely want today, in both the database and the network. It does a real-time sync, and we actually have event handling, so you can run code on the server and see responsive changes.

Of course, the real magic is in the automatic conflict resolution: the fact that you can adjust the changes. Why automatic conflict resolution instead of manual? The problem is if you do manual conflict resolution, you can never really do real-time, because things just stop. You really want to be able to do it at a real-time speed, so it has to be automatic.

You can actually change the scheme on the fly, you can push data to devices, and with the event frame you can introduce to an existing infrastructure. You have all these devices that all connect to the server. The server has a sync engine that makes all the automatic conflict resolution work. It has an event framework to run code on the server. There’s a dashboard, of course, so you can see what’s happening. There’s authentication, access control, and then you have all the Realms. Every single device can have its own Realm, and they can have Realms that are shared between all of them. You can have as many Realms as you want! That allows you to scale up so you can have millions of Realms, and you’d basically spread out your data around them. We can also connect to other servers, so if you have backend databases or APIs, you can actually make connectors to those. It will integrate into your existing infrastructure.

Live Objects

The key thing that enables this is the idea that everything is just objects all the way down. It’s objects, and the objects are live. If you use Realm already, you have this idea that even if you’re on multiple threads, you can change through multiple threads and you’re still safe, and you’ll see the updates in the other threads. We take that one step further and say that also happens with the network. You can have multiple devices and they all update the objects, but it’s always the same thing: they’re just live objects, they change. The objects are the data model, which really simplifies the mental model. It makes it so much easier to build and maintain apps. The idea is that you can define objects exactly in the same syntax you’d usually do it in.

You create these objects just like regular objects, but suddenly it’s kind of like supercharging them. You get the ability to do transactions and queries on them, you can do persistence, and you get all that for free. It all just works. Plus, it’s still full as a database, and it’s observable. With these features, this is the foundation that allows you to build pretty much anything you want in an app!

The Realm Mobile Platform: it’s available now, so download it! There’s a totally free version for developers: you can run it on your own hardware, and build anything you want with it.

We also have a commercial counterpart (the company has to live!) that solves enterprise needs and has more integration into your backend and more server-side handling, and custom authentication. But the free version is super powerful too! You can build full apps on it with no rate limits. You can use it for anything, as much as you want, for as long as you want.

Read more