Realm Blog

Why Can’t It Work Like Email?

by /

I was sitting at Starbucks when I overheard someone complaining about the usability, or rather the lack thereof, of an app on their phone. Clearly they were having network connectivity problems (all too often your phone will automatically connect to a wifi network but not actually be connected to the Internet until you process the click-wrap or portal-agreement page). At a moment of frustration, the individual exclaimed “I just want to use (the app)…why can’t it work like email!”.

This is a more astute comment than it might initially seem. For decades, email has been purposely designed to give the “right” user experience and properly function independently of network connectivity, latency, and available bandwidth – I can create, delete, and read emails locally on my device, and all changes are eventually sent or replicated back to the Cloud for action. These changes could be sent immediately, or could be delivered hours or even days later when my device is finally properly connected, but the main point is that the usage model for email is independent of the connectivity (at least for most email clients).

The usage model is clearly fundamentally different for most other mobile applications. Many actually block users from using the app if they are not properly connected to the network (see the screen shot below); others simply spin waiting for network connectivity, or otherwise do not allow users to perform updates and manage the data. While some apps may require real-time network access to properly function (e.g. real-time stock market quotes), the vast majority could actually “work like email” and provide a dramatically enhanced experience.

It shouldn’t have to be like this:network access image

Solutions like Realm allow developers to build their apps so that they do work like email. Realm is a new and improved way for managing the connectivity and data flow between Cloud and device, and, like email, ensures both that users can interact with the data locally on the device and that the changes (creations, deletions, modifications) are tracked properly and replicated back to the Cloud and other devices, either in real-time or whenever connectivity is restored. This new model enables a much improved user experience, which enables dramatically happier users, reduced churn, more revenue, and overall better results. Realm can be built into your existing Cloud service or accessed as its own SaaS service via the Realm Public Cloud. Come check it out, and join the growing community that is building better mobile applications.

Read more

It’s All About the User Experience Part III: Scalability and Improved Cloud-to-Edge Communication

by /

As I mentioned in part one (It’s All About the User Experience: Offline Support), and part two (Responsiveness and Zero-Latency Data Access), the Cloud has been an amazing enabler, allowing the rapid launch of new and innovative solutions and making those solutions and services accessible and available world-wide and from multiple devices. However, while the Cloud enabled and principally solved the any device, anytime part of the equation, the anywhere part assumes good connectivity and service responsiveness to be true, which often it isn’t. The Cloud, therefore, actually and is unfortunately degrading the user experience, especially relative to today’s users’ expectations of instantaneous experiences that are always available at one’s fingertips, everywhere and anywhere.

In part three and the final part of this series on the user experience, we’ll focus on the notion of scalability and improved communication between the Cloud and the true edge – the mobile or IOT device – and talk more about the connective tissue between the two.

Many applications today function by requesting data or various data sets from the Cloud, either on boot-up or based on specific user action. The data may include application content or meta-data, images and related graphics, A/B UI testing directives, or even fundamental aspects of the UX itself. However, most companies that I talk to find that as they grow in user adoption and thus the number of devices, they encounter a fundamental scaling problem – the size and amount of data, the sheer number of devices and the fact that this data needs to be exchanged over and over again strains the system architecture. The key interfaces (the “head-end”) in the Cloud that communicate with the devices begins to fall over under the strain, and the overall system no longer delivers the experience expected by the user community without costly and expensive internal system upgrades. It’s a fundamental problem caused by success, but that doesn’t make it any more palatable, especially since the cost of scaling the internal architecture can quickly outpace the financial gain of increased users and usage. The key to avoiding this scalability problem and its ensuing impact on the user experience is improving the communication between the Cloud and the devices at the edge.

Solutions like Realm bridge the gap between the device and the Cloud and allow for unprecedented efficiencies as solutions and applications scale. Rather than send all the data to the device every time, Realm allows developers to cache data on the device — Realm will update it automatically when it changes, which fundamentally reduces the amount of data that gets sent to the device as only updates need to be sent instead of sending all the data every time. Additionally, for large data sets, delta data diffs can be delivered instead of resending the entire object. These changes can dramatically improve the efficiency and scalability of the system, reducing overall load at the head-end of the Cloud, and not just make the system less expensive to operate but fundamentally deliver an improved user experience.

In a world where most services have multiple options, delivering the best possible user experience at the right price and cost structure is critical. The current default mechanism of fetching ever-increasing data sets from the cloud every time simply doesn’t scale, isn’t efficient, and ends up negatively impacting the user experience. REST is not actually best – solutions like Realm can help. Come and check out why various companies are coming to Realm to fundamentally drive the next-generation of user experience.

Read more

It’s All About the User Experience - Part II: Responsiveness and Zero-Latency Data Access

by /

As I mentioned in part one (It’s All About the User Experience: Offline Support),the Cloud has been an amazing enabler, allowing the rapid launch of new and innovative solutions and making those solutions and services accessible and available world-wide and from multiple devices. However, while the Cloud enabled and principally solved the any device, anytime part of the equation, the anywhere part assumes good connectivity and service responsiveness to be true, which often it isn’t. The Cloud, therefore, actually and is unfortunately degrading the user experience, especially relative to today’s users’ expectations of instantaneous experiences that are always available at one’s fingertips, everywhere and anywhere.

In part two of this series on the user experience, we’ll focus on the responsiveness of the user interface and the benefits of zero-latency data access.

As an example, in my home I have television and content services provided by a company with a Cloud DVR. In general, it works well, and I love the ability to access content from any device over any connection. However, during periods of high latency or lag, the solution becomes downright horrible, when even seemingly simple tasks such as browsing the content guide or changing the channel get queued up with no feedback for the user. The entire UI grinds to a halt, making the system unusable, and then at some point in the future it unfreezes and processes every queued key press in rapid succession, resulting in unanticipated behavior and frustrating results. While the content, of course, benefits from being in the cloud, the meta-data describing that content should be available on the device and shouldn’t necessarily be dependent on instantaneous cloud access because the user experience drops to “unusable” the instant the latency exceeds a given threshold.

Most mobile applications have the same problem – they rely on fetching data from the Cloud, and are largely unusable or otherwise paused when doing so. Latency cannot always be guaranteed on most networks, especially mobile and WiFi, but is critical to providing a good experience, driving usage, and even revenue. Amazon found every 100ms of latency cost them 1% in sales; Google found an extra half-second in search page generation time dropped traffic by 20%. When consumers have choices, they will use the solution that provides the best experience, and churn off the applications that don’t. So why don’t more applications focus on providing a good experience? The key to avoiding this latency-induced lag is storing data locally on the device and enabling zero-latency access to data.

Solutions like Realm allow developers to store data locally on the mobile/IOT device, keep it updated in real-time, and track changes both on the device and the Cloud to automatically resolve conflicts caused by updates. In the above example, the meta-data around content could be stored locally on my device(s) and updated as it changes (and in reality, it changes slowly). In the second example, the important data could be stored locally on the mobile or IOT device, ensuring that the UI is highly responsive and reactive, independent of the current connectivity characteristics. In this way the system can deliver unparalleled user experiences, ensuring continued usage, customer acquisition, revenue, and reputation.

In a world where most services have multiple options, and today’s expectation of service experiences (often assuming instantaneous access to data), delivering the best possible user experience is critical, and the current default mechanism of relying on on-demand data fetching from the cloud is not adequate to satisfy this requirement. REST is not actually best – solutions like Realm can help. Come and check out why various companies are coming to Realm to fundamentally drive the next-generation of user experience.

Read more

It’s All About the User Experience - Part I: Offline Support

by /

The Cloud has been an amazing enabler to allow the rapid launch of new and innovative solutions, and make those solutions and services accessible and available world-wide and from multiple devices. The Cloud is key for mobile-first experiences that are at the forefront of the digital transformation which many companies went through or are going through to enable any device, anytime and anywhere access. While the Cloud enabled and principally solved the any device, anytime part of the equation, the anywhere part assumes good connectivity, low latency, and service responsiveness to be true. While (rightly so) a lot of attention and investment is spent on UX and UI, the assumption for most is that connectivity issues and continuity of the experience under adverse conditions is relegated to backend engineering, or worse, the mobile carrier’s problem. Given that connectivity, bandwidth, and latency fundamentally change and are often incompatible with delivering a great experience, I might argue that the Cloud can actually and unfortunately degrade the user experience, especially relative to today’s users’ expectations of instantaneous experiences, always available at one’s fingertips, everywhere and anywhere.

In this multi-part series on the user experience, we’ll focus on various aspects of the user experience and how to improve it – first, let’s look at the user experience and the notion of offline support.

I have previously written about my experiences with airline apps on the mobile phone. It is highly frustrating to watch a mobile app spin while fetching data from the Cloud because the connection at the current point in time is not strong enough, suffers from high latency, or worse cannot get a data channel because all the nearby cell-towers or Wi-Fi hotspots at the Airport are overloaded. As a user, I understand the need to wait when I am fundamentally requesting new or updated data, but for data that rarely changes and should be instantly and predictably available, it can be incredibly aggravating.

Even worse, most applications today fundamentally don’t work when connectivity drops, often stalling or popping up “you are not connected” banners in the UI. While obviously certain transactions require connectivity, such as transactional services like purchases or state changes, most applications could continue to function during temporary network changes, perhaps in some degraded state, if the data being accessed was stored locally on the device. I call this offline support. And having offline support, I would argue, is critical to providing that seamless, good experience that mobile users expect. Whether you are browsing content tiles, accessing collections of data, or reviewing previous information and messages, having the data stored locally on the device and updated in real-time when connected to the network provides the best of both worlds – an up-to-date experience when connected to the network, and a useful, seamless experience when not. This is the way email has worked for the longest time – why shouldn’t all apps function this way?

Providing offline support is one powerful use case from Realm. Realm bridges the gap between the device and the Cloud and allow applications to provide the best experience – local data for offline support, the ability to keep that data updated in real-time, and the guarantee to track changes due to data updates and resolve any and all conflicts created by simultaneous updates. Realm allows developers to deliver an unparalleled user experience, independent of connectivity, by enabling this offline support.

In a world where most services have multiple options, and given today’s expectation of service experiences by the consumer (often assuming instantaneous access to data), delivering the best possible user experience is critical to driving revenue and usage, keeping and growing the user population, and minimizing churn. The current default mechanism of relying on on-demand data fetching from the cloud is not adequate to satisfy this expectation. REST is not actually best – solutions like Realm can help. Come and check out why various companies are coming to Realm to fundamentally drive the next-generation of user experience.

Read more

Technical Guest Post: “The things I’ve learned using Realm”

by /

It’s been exactly two years since I posted the article “How to use Realm like a champ”!

In this article, I try to sum up all the things I’ve learned from using Realm, and how it all relates with the current landscape of Android local data persistence: Android Architecture Components, Android Jetpack, Room, LiveData, Paging, and so on. Stay tuned!

An overview of Realm

Realm debuted as a mobile-first database, and has since grown into its own full-scale data synchronization platform. From the get-go, Realm’s vision was to provide the illusion of working with regular local objects, while in reality enabling safe and performant concurrent sharing of data. On top of that, Realm intends to handle most complexities of concurrency, and when something changes in the database, we receive notifications so that we can update our views, to always reflect the latest state of data. With the Sync Platform, these changes could be committed across multiple devices, and get seamlessly merged, like with any local write. Personally, I’ve used the Realm Mobile Database in multiple applications, and if there’s one major thing it taught me, it is to prefer simplicity over complexity. Sometimes, we might be so used to doing something in some way, that we don’t even think of just how much easier it could be with a slightly different approach.

In case of Realm, that different approach is observable queries, but of course, being able to define classes that are automatically mapped to database without having to think about relational mapping and joins is also a plus.

Out of the box, with Realm, pagination from a local data source is also not a real problem, as data is only loaded when accessed, therefore whether it’s 200 items or 10000, we can’t run out of memory, and the query will generally be fast.

A story of faulty abstractions

It is commonly seen as a good practice to abstract away specific implementations under our own interfaces, so that if need be, they can be safely replaced with a different implementation. And if you ever truly do need to swap out one implementation for the other, it’s definitely helpful that changes are localized to a single “module”, instead of it influencing parts of the code all over the codebase.

However, sometimes we think we’d like the module to behave one way, and enforce a contract that might not necessarily be the best solution to our problem. In this case, our interface restricts us from using a different approach.

In the case of local datasources, the common “mistake” is that if we have a DAO layer, then the DAO must have methods akin to this:

public interface CatDao {
   List<Cat> getCatsWithBrownFur();
}

In which case we’ve hard-coded that we have no external arguments (and is intended to be a constructor argument of the dao, most likely a singleton instance), and that retrieval of data is single, synchronous fetch. Following up on that could be, for example, a Repository:

public interface CatRepository {
   void getCatsWithBrownFur(OnDataLoaded<List<Cat>> callback);
}

In which case we’d ensure that the getCatsWithBrownFur() is most likely executed on a background thread, and will make its callback on the UI thread. The retrieval of data would be a single, asynchronous fetch. The data is loaded in its entirety to memory on a background thread, then passed to the UI thread for it to be rendered.

But is this really the best thing we can do?

What Realm’s queries look like

Observable async queries, change listeners, and lazy loading In Realm, or at least in its Java API, you receive a so-called RealmResults, which implements List. So we might be tempted to just use it as a List and move on. But that’s actually not how it’s intended to be used: Realm updates any RealmResults after a change has been made to the database, and notifies its change listeners when that’s happened.

So in practice, setting up a Realm query on the UI thread involves both defining the query, AND adding a listener to handle any future change that happens to it.

private Realm realm;
private RealmResults<Cat> cats;
private OrderedRealmCollectionChangeListener<RealmResults<Cat>> realmChangeListener = (cats, changeSet) -> { adapter.updateData(cats, changeSet);    // initial load + future changes
};
@Override
protected void onCreate(Bundle savedInstanceState) {
   super.onCreate(savedInstanceState);
   realm = Realm.getDefaultInstance();
     // ← open thread-local instance of Realm
   cats = realm.where(Cat.class).findAllAsync();
     // ← execute Async query
   cats.addChangeListener(realmChangeListener);
     // ← listen for initial evaluation + changes
}
@Override
protected void onDestroy() {
   super.onDestroy();
   cats.removeAllChangeListeners(); // ← remove listeners
   realm.close(); // ← close thread-local instance of Realm
}

For those familiar with Swift, the API is slightly different, but it does the same thing:

let realm = try! Realm()
let results = try! Realm().objects(Cat.self)
var notificationToken: NotificationToken?
override func viewDidLoad() {
   super.viewDidLoad()
   // …
   self.notificationToken = results.observe {
    (changes: RealmCollectionChange) in   // … (handle changes)
   }
   // call `notificationToken?.invalidate()` when needed
} ```

But we’re getting lost in details. What’s different here, compared to the aforementioned common DAO approach? Well, pretty much everything.

The common DAO pattern assumes that we want to call the DAO to refresh our data, whenever we know that any other parts of the application, on any other thread could have potentially changed the data we’re showing. That’s quite the responsibility!

In case of Realm, we can observe the query results — in fact, Realm automatically updates it for us, whenever it’s been changed. In this case, our listener is called: and we even receive the positions of where items have been inserted, deleted, or modified. When we commit a transaction, Realm handles all of this for us in the background.

Another thing to note is that when using Realm’s Async queries, the evaluation of the new result set and the diff happens on Realm’s background thread, and we’ll see the new data along with the changes only when the change listeners are being called.

When we call an accessor on a RealmResults, we actually receive a proxy instance, that can read from and write to the data on the disk, essentially minimizing the amount of data read to memory. This eliminates the need for pagination, even when working with larger datasets.

RealmResults<Cat> cats = realm.where(Cat.class).findAll();
  // sync query for sake of example
Cat cat = cats.get(0);
  // actually `fully_qualified_package_CatRealmProxy`
String name = cat.getName();
  // calling accessor reads the data from Realm

The price of proxies: threading with Realm

Anyone who’s used Realm knows that there is a price to pay with proxy objects and lazy results, which also extends to Realm’s thread-local instances. Namely, that while Realm handles concurrency internally, the objects/results/instances cannot be passed between threads.

final Realm realm = Realm.getDefaultInstance();
Thread thread = new Thread(new Runnable() {
@Override
public void run() {
 RealmResults<Cat> cats = realm.where(Cat.class).findAll(); // <--
// IllegalStateException: Realm access from incorrect thread.
// Managed objects, lists, results, and Realm instances can only be
// accessed on the thread where they were created.
}
}).start();

I’ve often heard the claim that “threading with Realm is hard”. I firmly believe this claim is false.

Even when there are multiple threads involved, using Realm is pretty easy — just get an instance of Realm for while you need it, then close it.

// on a background thread
try(Realm realm = Realm.getDefaultInstance()) {
   // open Realm for thread
   // use realm
} // realm is closed by try-with-resources

The one special case the developer has to remember is that instead of passing a managed object, you should generally pass its primary key, then re-query it on the background thread instead.

final String idToChange = myCat.getId();
// create an asynchronous transaction
// it will happen on background thread
realm.executeTransactionAsync(bgRealm -> {
// we need to find the Cat we want to modify
// from the background thread’s Realm
Cat cat = bgRealm.where(Cat.class)
         .equalTo(CatFields.ID, idToChange)
         .findFirst();
// do something with the cat
});

However, that’s actually as far as complexity goes, when it comes to Realm and threading, out of the box. RealmResults are already evaluated automatically by Realm on a background thread, and then the data is passed to the listener on the UI thread, so there is zero reason to ever try to pass them between threads.

Why people think threading with Realm is hard

So why is the claim so common? After spending a lot of time on the realm tag on Stack Overflow, I believe it comes down to the following primary causes:

1.) The developer has introduced RxJava into their project without understanding it, and end up introducing such level of complexity that they don’t understand what threads their code is running on.

2.) The developer doesn’t know how RealmResults works, specifically that findAllAsync() will evaluate the results on a background thread, and then the RealmChangeListener will receive it on the UI thread — and instead they attempt to handle the threading that Realm would already handle for them internally.

3.) Reluctance to pass instance of Realm as a method argument, and/or intending to use Realm’s thread-local instances as global singleton.

The first and second causes are self-explanatory. But what about the third?

When you “get an instance of Realm” with Realm.getDefaultInstance(), you actually receive a thread-local and reference counted instance, where the reference count is managed by calls to getDefaultInstance() and close(). It is not a singleton, and it cannot be accessed on different threads.

It’s a little known fact, but to avoid passing Realm and instead see it as a thread-local singleton, it’s possible to store it in a ThreadLocal. Then, once set on a given thread, Realm can be acquired from it as simply as Realm realm = threadLocal.get(). This is one way one could turn instances of Realm into globally accessible variables. In this case, what we need to manage is Realm’s reference counting.

Why do people need and want a singleton instance of Realm? To create the data layer implementation for their abstraction, of course!

public interface CatDao {
List<Cat> getCatsWithBrownFur();
 // ← initial attempt: uses List (no changes!) and has no arguments
// -------------
// alternatives?
RealmResults<Cat> getCatsWithBrownFur();
 // ← problem: Realm opened and closed could immediately
 // invalidate the results, Realm-specific

RealmResults<Cat> getCatsWithBrownFur(Realm realm);
// ← problem: We’re passing Realm as a dependency
 // as part of the contract, Realm-specific
}

So we’d like to create a contract that lets us hide Realm as an implementation detail, but takes it into consideration that data could be evaluated on a different thread, and is sent to us with some delay through a subscription (to a listener). And on top of that, any future writes to the database that invalidate our current dataset should trigger a retrieval of the new data, and notify us of change.

public interface CatDao {
  LiveObservableList<Cat> getCatsWithBrownFur();
  // it would be possible to write a wrapper
  // around RealmResults like this
  Observable<List<Cat>> getCatsWithBrownFur();
  // it is possible to convert listeners to Rx Observable
  // something else???
}

How Realm shaped the future of local data persistence on Android

The introduction of Jetpack, Room, and LiveData With time, other libraries emerged that supported the notion of “observable queries”. One of the first to follow Realm was SQLBrite, which exposed SQLite queries as Observable<List> — and used RxJava to do so.

Eventually, Google created their own approach, called the Android Architecture Components — which are now a part of Android Jetpack. The idea was to solve common problems that developers face when writing applications, and simplify them by providing a set of libraries and an opinionated guideline that helps solve these problems.

Observable queries with Room and LiveData The most notable addition to the Architecture Components is LiveData. It is a “holder” that can store a particular item, and can have multiple observers. Whenever the data stored within the LiveData is changed, then the observers are notified, and they receive the new data. When a new observer subscribes for changes of the LiveData, it receives the previously set latest data.

One of the biggest additions to the Architecture Components was Room, which is Google’s own ORM over SQLite. But what’s more interesting is that it allowed defining a DAO (for accessing the entity’s tables) like this:

@Dao
public interface CatDao {
  @Query(“SELECT * FROM CATS WHERE FUR = 'BROWN'”)
  LiveData<List<Cat>> getCatsWithBrownFur();
}

We can then use this in our Activity:

private LiveData<List<Cat>> cats;
private Observer<List<Cat>> observer = (cats) -> {
adapter.updateData(cats); // initial load + future changes
};
@Override
protected void onCreate(Bundle savedInstanceState) {
 super.onCreate(savedInstanceState);
 CatDao catDao = RoomDatabase.getInstance()
                    .catDao(); // get Dao
 cats = catDao.getCatsWithBrownFur(); // ← execute query
 cats.observe(this, observer);
// ← listen for initial evaluation + changes
}
@Override
protected void onDestroy() {
super.onDestroy();
 // no need to unsubscribe,
 // because of `observe(LifecycleOwner`
}

Doesn’t this look extremely familiar? Swap out Observer for RealmChangeListener, and LiveData<List> for RealmResults, and it should look almost just like the example above!

When we subscribe for changes of a LiveData (or more-so, “start observing it”), then as there is at least one active observer, Room begins to evaluate the query on a background thread, then passes the results to the UI thread. Afterwards, Room tracks the invalidation of the table(s) this query belongs to, and if that table is modified, the query is re-evaluated.

The key differences are that the diff calculation is moved to the adapter (see ListAdapter), and that LiveData’s lifecycle integration allows for automatic unsubscription of its observers, instead of having to do it explicitly.

Otherwise, the behavior is rather similar, in fact, so similar that LiveData<List> is a possible way to wrap RealmResults as merely an implementation detail — as it’s also shown in Realm’s examples.

The downside of LiveData<List>

In case of Room, there’s a downside to using LiveData<List> directly with large datasets. Each time a new version of the dataset is retrieved, the full dataset is copied to memory. So assuming we have 10000 cats, and modify a single cat, all other 9999 cats will be retrieved from database again as well in the newly evaluated list.

When relying on Realm’s lazy evaluation, this isn’t really a problem: we only ever retrieve items when we access them. We don’t load the full dataset to memory.

However, Google realized this poses a problem, and began working on a solution. The release of a new Architecture Component: Paging To eliminate the need for complete re-evaluation of a modified dataset each time a change occurs, Google invented something amazing: the LivePagedListProvider. Since then, the API has changed a bit, so it’s more appropriate to refer to it as the combination of DataSource.Factory and LivePagedListBuilder. With help of these classes, it’s possible to expose an observable query as a LiveData<PagedList>.

A PagedList is like a list, except it is backed by a DataSource, and it only loads pages of data at a time, instead of the whole dataset. If an item is accessed that’s not yet loaded, then that page is loaded from the datasource on a background thread. When the page of data is loaded, it’s passed to the UI thread, and can be “seamlessly diffed” into the currently loaded dataset. For this to work, Google also provides the PagedListAdapter.

The way Room’s Paging integration works is that we can expose a DataSource.Factory instead of a LiveData<List>.

@Dao
public interface CatDao {
 @Query(“SELECT * FROM CATS WHERE FUR = 'BROWN'”)
 DataSource.Factory<Integer, Cat> getCatsWithBrownFur();
}

Then we can do: private LiveData<PagedList> cats; private Observer<PagedList> observer = (cats) -> { pagedListAdapter.updateData(cats); // initial load + future changes };

@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
CatDao catDao = RoomDatabase.getInstance()
                    .catDao(); // get Dao
DataSource.Factory<Integer, Cat> dataSourceFactory = catDao.getCatsWithBrownFur();
cats = new LivePagedListBuilder<Cat>(dataSourceFactory, new PagedList.Config.Builder()   .setPageSize(20)   .setPrefetchDistance(20)   .setEnablePlaceholders(20)   .build())
.setInitialLoadKey(0)
.build();
cats.observe(this, observer);   // ← listen for initial evaluation + changes
}

This allows Room to expose an observable query, where the data is fetched asynchronously, but page by page, instead of loading the full dataset. This allows Room to be much faster than using regular SQLite, while retaining the benefit of observable queries.

Making Realm work with Paging

Realm already brings lots of benefits to the table, but many people intend to use Realm as an implementation detail, not directly. As such, they might want to keep observable queries (just like Room does), but don’t want to use managed results and managed objects: they always see the latest state of the database, which also means they mutate over time. They’re still proxies, and not regular objects.

One possibility would be to read the data from Realm using realm.copyFromRealm(results). However, this method could only be called on the current thread (in this scenario, generally the UI thread), and it would read the full dataset from disk. We’d have the exact same problem as with LiveData<List>, and would be reading large datasets at once on the UI thread! This is clearly a terrible idea, so what’s a better alternative?

We could move the copying of data from Realm to a background thread, but Realm cannot observe changes on regular threads, only on threads associated with a Looper. However, it’s possible to create a HandlerThread — if we can create a Realm instance on this handler thread, execute and observe queries on this handler thread, and keep these queries (and the Realm instance) alive while we’re observing them, then it can work!

Even then, we would still have the problem of loading the full dataset on the handler thread for each change. However, not if we can make Realm’s query results be exposed through the Paging library! Monarchy: global singleton Realm with LiveData and Paging integration I’ve been working on a way to expose RealmResults in an observable manner from a handler thread, either copying the dataset, mapping the objects to a different object, or through the Paging library.

The end result is called Monarchy, a library that lets you use Realm very similarly to as if you were using LiveData for a Room DAO. To create a global instance, all it needs is a RealmConfiguration, and then we can use it as a DAO implementation like this: public interface CatDao { DataSource.Factory<Integer, Cat> getCatsWithBrownFur(); } @Singleton public class CatDaoImpl implements CatDao { private final Monarchy monarchy; @Inject CatDaoImpl(Monarchy monarchy) { this.monarchy = monarchy; } @Override public DataSource.Factory<Integer, Cat> getCatsWithBrownFur() { return monarchy.createDataSourceFactory(realm ->
realm.where(Cat.class) .equalTo(CatFields.FUR, “BROWN”) ); } }

One strange design choice of the Paging library is that the fetch executor can only be set on the LivePagedListBuilder, and not on the DataSource.Factory. This means that to provide Monarchy’s executor that executes fetches on the handler thread, Monarchy must be used to create the final LiveData<PagedList>:

LiveData<PagedList<Cat>> cats = monarchy.findAllPagedWithChanges(
   dataSourceFactory, livePagedListBuilder);

On the other hand, it is still quite convenient: we receive unmanaged objects that are loaded asynchronously, page by page, and we’ve also retained the ability to receive future updates to our data when the database is written to.

Another interesting tidbit is that Monarchy uses LiveData, therefore depending on whether there are active observers, it can automatically manage whether the underlying Realm should be open or closed: completely moving lifecycle management of Realm instances into the onActive/onInactive callbacks of LiveData.

Conclusion

If it weren’t for my time using Realm, it would probably be much harder for me to understand the driving forces that shaped the Android Architecture Components: especially Room, and its LiveData integration.

The ability to listen for changes and always receive the latest state of data with minimal effort just by subscribing to an observable query is something that at first might seem foreign, but simplifies the task of “fetching data from the network, storing it locally, and also querying for and displaying data” — which is what most apps need to do.

Why would we manually manage data load callbacks, if we could just run background tasks that fetch data from network and write it directly into our database, while all we need to do for the UI is observe for changes — and have any new data passed to us automatically?

What if in “Clean Architecture”, fetching data isn’t even a “use-case”, but just an Effect of finding that our data is outdated and should be refreshed?

What is the point of introducing Redux, if it restricts us from making subscriptions to our database queries, as the database would become a second store, and therefore would force us to make data loading be a single fetch inside “middlewares”? (Unless subscriptions are seen as a “state” that should be built up as a side-effect in the store’s observer, of course).

What if the abstraction we’re trying to build binds our hands and keeps us from finding simpler solutions that solve our problem in a more efficient way?

Realm’s observable queries were ahead of its time, but shaped the future of Android local data persistence. Instead of manual invalidation of data, we could just observe for changes. What else is there that we take for granted, build on top of, and keeps us from finding a better solution?

Using Realm taught me that even though there’s a common way of doing things, sometimes taking a completely different approach yields much better results. I’m glad that I had the chance to try Realm, and could learn from the opportunity.

Note from Realm:

We liked Gabor’s article, thus we want to re-post it here with Gabor’s permission. Gabor Varadi regularly writes articles on Medium which can be found here.

While Monarchy is not an official Realm library, it is an interesting complimentary library developed by Gabor. You can find it and file issues on his repo (https://github.com/Zhuinden/realm-monarchy)

Read more

5G is coming: Faster – But What Else is Needed?

by /

There has been a lot of talk and perhaps hype about 5G in the wireless industry, with arguably one of the largest announcements being between T-Mobile and Nokia of a $3.5B infrastructure investment. 5G delivers higher bandwidth and is better suited for IoT and mobile by supporting the anticipated explosion of devices with smarter allocation of spectrum, lower latency, higher bandwidth and improved power handling. But I’ll argue that 5G only addresses part of the solution gap, and omits a key element that has been historically neglected. With each new generation of wireless network and the promises surrounding it, the industry underplays that by their very nature, mobile and IoT devices are fundamentally semi-connected: sometimes the connection is unavailable, sometimes it’s adequate but sub-optimal, and other times it works great. Even worse, users cannot always predict which situation they will be in, for any given location and time of day. Improved coverage through mid-band LTE (as per T-Mobile US), and augmenting coverage with WiFi and CBRS, helps of course, but we’ve all experienced applications and services that simply didn’t work properly and provided a poor experience when the mobile device was disconnected from the network, or in a semi-connected state with high-latency. This situation – the fact that coverage is not universal, and not universally consistent across geographies and even time of day – is a fact of life and fundamentally will not change.

Both proponents of 5G and IoT talk a lot about computing moving to the edge, and the advantage for applications when doing so. Exceptional consumer experiences are only obtained from immediate, i.e. zero-latency, access to relevant data; in fact, many applications fundamentally require immediate access to data or else risk loss of functionality, decreased revenue, failure of business operations, or consumer churn. For many of today’s connected applications, data must be immediately accessible at the edge – the true edge, such as one’s phone, wearable, sensor, car, etc. While mobile networks can sometimes deliver low-latency connections, and that is a key promise from 5G, the network conditions can never be guaranteed for devices that fundamentally move in time and space. The “always connected”, low-latency, mobile network does not exist; rather, the application developer and service architect must overcome this obstacle to reduce risk around loss of revenue, customers, functionality, and correct operations.

The fundamental way to ensure zero-latency access to data is to put the data on the device, prior to use, and keep it synchronized and updated behind the scenes. In this way data is guaranteed to be accessible at all times, can be processed and updated independent of the real-time connectivity dynamics, and the consumer or business application operation can complete successfully, with changes updated automatically once connectivity is restored. Especially in a competitive market where consumers have multiple options, they will inevitably flock to the solutions that provide superior mobile experiences. Similarly, business applications – mobile applications, process flows, field services, or IoT – need data at the edge to ensure business continuity, revenue, efficiency, and correct operations.

Realm solves this problem through its world-renown mobile-client database and its synchronization companion (Realm Platform) for both businesses and consumer use-cases. Starting from its roots as a developer-friendly, mobile-optimized solution, Realm allows data to be cached or synchronized at the true edge for mobile, IoT, and connected devices, ensuring that the data is kept up-to-date and that conflicts are reliably resolved. The Realm Platform can be flexibly deployed in both public and private Clouds to support various use-cases, and can integrate with any external database or data source to effectively mobilize and synchronize any set of data.

5G promises to move the connected world forward for faster and better user experiences, more connected devices (including IoT) and higher bandwidth. But it does not overcome the reality of the mobile world: being disconnected. For optimal consumer experiences and true business success, applications and services must address connectivity changes and occasional disconnectivity by putting the data at the true edge. The success of mobile, IoT, and connected applications, whether on 5G or other, requires Realm and its seamless data synchronization capabilities for a true fulfillment of the promises that 5G and the future intends to provide.

Check out Realm at www.realm.io or sign up for a free trial with our public Cloud at cloud.realm.io.

Read more

Building Modern Swift Apps with Realm Database

by /

We love that the Realm development community at large is so, so good at taking existing code and content and iterating on it to make great things even better. That’s why we’re pleased (and humbled!) to let you know about a new book that covers Realm development in the iOS world: Read more about it here.

Why a book about Realm? Sure, the Realm API docs are quite complete, and the tutorials from the Academy do a great job at giving you hands-on experience at integrating Realm into your apps. But the nice thing about this book is that it pulls all of those concepts together in one place, and takes you all the way from foundational concepts to building out multiple apps that leverage Realm just as you would in your real-world projects.

Author Marin Todorov recognized early on that there just wasn’t a good, comprehensive resource for “all things Realm” — so he wrote his own resource. That eventually turned into the book “Realm: Building Modern Swift Apps with Realm Database”, which, as far as we know, is the only complete book about developing modern Realm apps against iOS.

“Finally, a book that doesn't feel like homework! Right off the bat, the book shows you how to add basic Realm capabilities to your app. The further down the rabbit hole you go, the more advanced Realm features you'll discover.” — Gabriel Rosinski, iOS Developer

It’s easy to throw around numbers and stats, like our installed product base, the ever-growing number of developers who have adopted Realm in their projects, or even the stars on our GitHub repo. But when someone cares enough about your platform to sit down at their desk, put the digital equivalent of pen to paper, and write an actual book about your product to share with other developers? That’s humbling, and shows us that what we’re doing here at Realm is really making a difference in your day-to-day development.

In this book, you’ll take an app-centric view of working with the various features of Realm in real-world app scenarios. You’ll cover common use cases with the platform; dig into schemas and relationships; learn about the built-in notification APIs; manage Realm configurations; handle dependency injection and testing; and implement some solid migrations strategies that all showcase the great features that Realm has to offer the modern app developer.

“Marin and the RW team wrote the ultimate Realm guide. From the basics: how it differs from Core Data, how to model the data, and have the UI reacting to changes; up to more advanced scenarios: syncing data with the Realm Cloud and multi-threading. This book covers everything an iOS developer needs in order to write simple or complex apps using Realm, and I totally recommend it!” — Natan Rolnik, Kik

And as a bonus, the final chapter in the book is a great resource for getting up and running with Realm Cloud with a minimum of fuss. It covers data access, synced Realm providers, partial sync, how to manage your sync subscriptions, and more. Although talking about Realm Platform could easily fill its own book, this bonus chapter has everything you need to get started with real-time sync.

The book is from the folks over at http://www.raywenderlich.com, who also have some great written and video resources about Realm. As part of the launch event around this book, they’re offering the book to the Realm community at a discount for a limited time. We’d suggest that you head over and grab the book while the discount is still live; it ends on Friday, April 27. You can get the book here:

Read more about it here

We’d love to know what you think of the book. If there’s something you think that could benefit from being covered in a book like this, let us know and we’ll pass it along!

Read more

Realm Announces Product Name Changes and Sunsetting of Developer Edition

by /

Today Realm announces it will be sunsetting the “Realm Platform Developer Edition” as well as merging our “Enterprise” and “Professional” Editions under a new product name “Realm Platform Standard”. For any of our existing “Professional” and “Enterprise” customers there will be no changes to your existing agreements, but for our “Professional” customers you may have the ability to make use of additional features by contacting us. While the “Developer” Edition has been a great way for developers to experience a limited set of features of the Realm Platform software, our research has shown that the vast majority of users need our Enterprise features. To this end we will now offer an extended trial period on “Realm Platform Standard” edition for early development and next month we will be releasing a 30-day free trial for Realm Cloud as well. Lastly, we have created a much more scalable pricing model for both the cloud and self-hosted versions that will allow you to get started at the right price and have Realm scale with your business.

Starting today (February 26, 2018) the Developer Edition will no longer be offered for download. Don’t be concerned…we will continue to support any existing deployments for critical defects until April 30, 2018 and will advise you on migrating to Realm Platform Standard. If you have any questions please email us at: [email protected]

Frequently Asked Questions:

• What does the Developer Edition Sunsetting (February 26, 2018) mean to me ?

Developer edition was designed for developers to get access to a minimal set of functionality in Realm Platform that they could use in development and in the early stages of the apps release. After careful consideration and a great amount of feedback from our users, we will no longer develop, maintain, or release the Realm Platform Developer Edition after April 30, 2018. We also will be combining the various software editions into one new edition called Realm Platform Standard. This version will essentially have all the functionality of our old Enterprise edition in one, more economical edition. With this new edition (offered in both the cloud and self-hosted deployment models) we will also provide a free trial version in the cloud that has all the core features and no limitations on functionality, as well as an extended trial period for developers to use and develop with the self-hosted version, again with no limits on functionality.

•How do I migrate from Realm Developer Edition to Realm Platform Standard?

Please contact us at [email protected] and we can guide to the best path forward.

•What is the detail on the new pricing models going forward so that I can get started at the right price and have Realm scale with my business?

The new pricing models will be updated on our website shortly. In the interim, if you have any pricing questions, please feel free to contact us at [email protected]

•Can I get access to the Realm Platform Standard on the Cloud now?

Yes. We have the service up and running today in beta, with an expected GA date soon. You can sign up for the beta service at https://cloud.realm.io/

•What’s the best way for me to contribute my feedback and desires for future functionality in Realm Platform Standard?

We appreciate your feedback. Please email us at [email protected]

Read more

See Realm at INDEX Developer Conference

by /

The Realm team is headed to INDEX San Francisco on February 20-22, 2018. We are excited to join the open developer community and catch up with the latest on Mobile, IoT, AI, Microservices, Serverless and more.

Come say hello!

Stop by our booth at the expo! We’ll have lots of Realm goodies to hand out. This is also a unique opportunity to catch a hands-on workshop by our Engineers Ian Ward and Matthew Geerling. Together they will lead participants in building an offline-first mobile app using the new Realm Cloud.

Workshop details
“Building an Offline First Mobile Application using the new Realm Cloud”
12:00 PM-02:30 PM | Tuesday, Feb. 20 Moscone West/Level 2, Room 2002

In addition, Realm CEO and co-founder, Alexander Stigsen, will be demonstrating the latest addition to the Realm family: Realm Cloud. As part of the demo, he will be showing how to build a multi-user To-Do list tracking app on iOS that takes advantage of this new service — from provisioning to working app — in under 15 minutes.

Presentation details:
“Realm: from zero to scalable infrastructure in 60 seconds”
12:05 PM-12:20 PM | Wednesday, Feb. 21 Moscone West/Exhibit Hall, Theater

Don’t have a ticket yet? Get in touch with us to get a 20% discount on your conference ticket. Please email Hernando Buitrago at [email protected]

Want to chat one-on-one with our team during the conference? Please email Jeff Olson at [email protected]

realm at Index share image

Read more

Announcing Realm Cloud: Realm Platform as a Service Beta

by /

Today we’re announcing Realm Cloud, the new “as-a-service” version of Realm Platform. Realm Cloud brings mobile developers the easiest way to add offline first and realtime data sync features to their apps. Our product vision revolved around one word: simplicity. Hundreds of thousands of developers around the world have built Realm into their Android and iOS apps on the device side; we wanted Realm Cloud to be so simple that all of those developers could use it as an organic extension of those apps.

We’re launching it today as a limited early beta, and have opened a beta waitlist that will allow us to invite sets of new developers every week. Sign up today – our first 500 active beta users will be our “Charter Users,” eligible for discounts and Realm swag.

Why Realm Cloud, and Why Now

We launched Realm Platform as an on-premise product exactly a year ago this month. And we’ve spent most of the last year working with mid-to-large sized customers who both needed a realtime, offline-first data platform like Realm, but who also had server-side teams and a preference for an on-premise implementation. Making these customers successful – you can read some of their stories here – taught us a lot, and today we have an on-premise product that is much more scalable, reliable, and developer-friendly. It will only continue to improve going forward, as we are committed to these customers and others for whom a cloud service is not a fit.

Along the way, though, we’ve had many, many conversations with startups, smaller companies, and smaller project teams at big companies who loved what Realm Platform could do, but couldn’t use it. They got stuck on either the hassle of a server-side implementation or the pricing model, which was optimized for established commercial apps and larger customers.

Today, we’re pleased to finally have the answer in Realm Cloud. Getting started with Realm Cloud is easy, the user experience is intuitive and designed to work seamlessly with Realm Studio, and you can have your first realtime app up and running in about ten minutes. No server-side anything required – just sign up and get going. And while at this early beta stage we are not quite ready to announce pricing and packages, we can tell you that, when we are in production, getting started on Realm Cloud will be free, with a monthly pricing model that starts at a low enough amount to easily put on a credit card, and scales with you and the needs of your app. (The beta of course is totally free.)

How it All Works, and How to Get Started

When we first started planning Realm Cloud, our goal was to do more than just automate the operations of Realm Platform. Instead, we wanted to make sure that the ease of use extended all the way into our APIs. To do so, we added a new synchronization mechanism: query-based sync.

Since the Realm Platform launch, our APIs were straightforward: provide a sync configuration with a URL and User object to open a synchronized Realm.

// Create the configuration
let syncServerURL = URL(string: "realm://localhost:9080/~/userRealm")!
let config = Realm.Configuration(syncConfiguration: SyncConfiguration(user: user, realmURL: syncServerURL))

// Open the remote Realm
let realm = try! Realm(configuration: config)
// Any changes made to this Realm will be synced across all devices!

The data synced to the Realm in the client device would always be kept up-to-date with the copy of the Realm in the Realm Object Server. This API was familiar to Realm developers since it mirrored the configuration when using the database on its own. This simplicity, however, also introduced a constraint. Your application data needed to be segmented into several private or shared Realms.

For many data models this was straightforward, however, others didn’t map as well… until now.

Query-based sync allows a developer to use Realm’s query engine to define what subset of data from a synced Realm is needed by the client device. This enables applications to cache only the data needed by the user and more efficiently work with shared data across users.

Using this new functionality is easy, simply enable it in the sync configuration. Next, perform a query on the Realm and then subscribe to that query:

// Create a partially synced Realm
let config = Realm.Configuration(syncConfiguration: SyncConfiguration(user: user, realmURL: realmURL, partial: true))
let realm = Realm(configuration: config)

// Create an anonymous subscription
let adults = realm.objects(Person.self).filter("age >= 18").subscribe()

The server will run the query and then synchronize all objects that match it. This query will be maintained for the life of the subscription and when data changes the server will re-evaluate the query and synchronize any changes. Given Realm’s reactive architecture, the app can stay-up-to-date by using the existing notification APIs:

// Normal change notification
self.notificationToken = adults.observe { (changes: RealmCollectionChange) in
    switch changes {
    case .initial:
        self.tableView.reloadData()
        break
    case .update(_, let deletions, let insertions, let modifications):
        self.tableView.beginUpdates()
        self.tableView.insertRows(at: insertions.map { IndexPath(row: $0, section: 0) }, with: .automatic)
        self.tableView.deleteRows(at: deletions.map { IndexPath(row: $0, section: 0) }, with: .automatic)
        self.tableView.reloadRows(at: modifications.map { IndexPath(row: $0, section: 0) }, with: .automatic)
        self.tableView.endUpdates()
        break
    case .error(let err):
        fatalError("\(err)")
        break
    }
}

The application can perform as many queries as needed. The sync protocol will normalize them and only send the minimal operations needed to keep the client synced. If the data is not needed anymore, you can choose to unsubscribe and the cached data for that query will be removed from the device:

// Remove the anonymous subscription
realm.unsubscribe(adults)

// Create a named subscription
let adults = realm.objects(Person.self).filter("age >= 18").subscribe(named: "Adults")

// Remove a named subscription
let didRemove = realm.unsubscribe("Adults")

We are really excited about the new query-based sync and will be sharing more in the coming weeks, including the support for finer-grained object-level permissions that compliment it. We would love your feedback, so be sure to sign up for the Realm Cloud beta to be one of the first to try it out!

Be a Charter Realm Cloud Developer

Register for the Realm Cloud Beta today – with limited capacity at this early stage, we’re inviting developers from the waitlist on a first-come, first-served basis. And if you are one of the first 500 users to get a Realm Cloud app up and running, you’ll get a 50% discount on Realm Cloud basic for its inaugural year of service. And if you give us useful feedback on Realm Cloud during the beta period, we’ll reward you with exclusive Realm swag like stickers, t-shirts, and water bottles.

Read more

Realm Scales to the Web with GraphQL

by /

In the last 7 years, Realm has been focused on perfecting the solution to a problem that mobile engineers and companies have faced for years: how to store data efficiently on the device and reliably receive and push data into the cloud.

Realm Middleware for Web

Realm’s offline-first approach to synchronization has eliminated many of the dependencies and issues that developers had previously had to deal with. Now, Realm is expanding their offering to the web! 🎉

Realm + GraphQL

After we launched Realm Platform, we heard our customers requests for front-end web support. Traditionally building a realtime app using Realm on the web has been a challenge. You can use realm-js or realm-dotnet to incorporate Realm data into a backend web server, but front-end work in the browser was not possible due to environment limitations.

In order to deliver Realm data to web frameworks, we pursued two approaches. First, we explored building traditional REST and Websockets APIs. Secondarily, we experimented with GraphQL, utilizing it in the 1.0 Realm Object Server web dashboard. In comparing the two efforts, we realized that GraphQL was a much better fit. Realm’s schema model is highly compatible with that of GraphQL. Realm is an object-database which supports links as a first-class type. This allows developers to build complex object-graphs especially because Realm uses advanced memory management to efficiently traverse a graph in their apps. However, this breaks down with a web API. When you perform a GET request to query for data in a Realm, how do you know what part of the graph to return?

With a traditional REST API you can choose to return linked objects through keys in the JSON, which results in making multiple requests to retrieve the data needed from the graph. We also explored adding a depth parameter to allow the request to specify which linked data to return.

Ultimately, this discussion started to look more and more like GraphQL!

The GraphQL query language allows developers to define what data they want returned in the API. This works off a defined schema, mapping perfectly to the Realm object model, and results in collapsing of multiple traditional REST API requests. Furthermore, the GraphQL spec also includes the concept of subscriptions allowing a developer to subscribe to realtime data events.

Putting it all together it became obvious that Realm was a great fit for GraphQL and vice-versa. As a result, today we are announcing support for GraphQL in Realm Object Server. This brings similar expressive realtime data access that Realm Database SDKs provide for mobile to the web.

Terse, Simple, Elegant

GraphQL offers a very terse, yet expressive way to interact with your backend.

const query = gql`
  query {
    companies {
      companyId
      name
      address
    }
  }`;
 
const response = await client.query({
  query: query
});

const companies = result.data.companies;
const mutation = gql`
  mutation {
    result: addCompany(input: {
      companyId: "some-unique-id"
      name: "My Amazing Company"
      address: "Mars"
    }) {
      companyId
      name
      address
    }
  }`;

const response = await client.mutate({
  mutation: mutation
});

const addedCompany = response.data.result;

Simply define queries, mutations.

Efficient

One of the problems inherent to REST is that you’re dealing with Resources. This means that even if you only need a portion of data to populate your web UI, you still need to pull the entire resource.

Let’s take a look at an example.

const UserSchema = {
       name: 'User', 
       properties: {                
               userId: 'string',
               firstName: 'string', 
               lastName: 'string', 
               birthday: 'date',
               accounts: {type: 'Account[]'}
       } 
}; 
const AccountSchema = { 
     name: 'Account', 
     properties: { 
             AccountId: 'string',
             name: 'string', 
             openDate: 'date'
     } 
};

If you were building a view where you needed the user’s, firstName and userId to display. You would still need to fetch the entire user (e.g.GET /users/{id}) to get those pieces of data. The user may have a dozen other properties that are not important for this page.

Furthermore, if you wanted to pull back the name of the top 3 accounts for each user, you’d need to make additional REST calls to get each object, just so you could reference the name of each.

One typical workaround for this is to create Composite Resources for this (e.g. UserAccountSummary). While this addresses efficiency concerns in bandwidth and speed, it also leads to potential data redundancy or additional code on your server end to munge this data together.

GraphQL queries solve this problem quite elegantly by allowing you to specify only the data you need.

const query = gql`
  query {
    users {
        firstName
        userId
        accounts { 
            name 
        }
    }
  }`;

Notice that this query is only fetching the specific fields it needs for the client, not only is this extremely efficient for speed and bandwidth as we’ve discussed. It’s also simpler for developers to read. It reduces the cognitive load of reading the code, seeing only what data is being consumed on the client and nothing more.

Powerful Realtime Subscription model

The GraphQL subscription model is a perfect fit for Realm’s live, reactive nature.

const subscription = gql` // Subscribed to updates.
  subscription {
    users {
        firstName
        userId
        accounts { 
            name 
        }
    }
  }`;

Notice, by simply changing the query to a subscription, updates to the underlying data will automatically synchronize from Realm to the client in real-time. There is literally nothing else the developer has to do to make all of this work. Notice that just as with mobile, Realm Web, also will trigger an update if any updates to the underlying database affect the results. This all creates a simple, intuitive, experience for developers.

Thriving Community

GraphQL is a proven technology and has a thriving developer community with numerous support channels from forums and Slack groups to videos and blogs.

The Future

GraphQL and Realm is a great pairing for the web, and the next step toward building reactive, data centric applications on the web. You can try it out for yourself by running the Realm Object Server w/ GraphQL, available here. We’ve also created a sample client here. We hope you’ll check it out and give us your feedback!

Cheers! 🎉

The Realm Team

Read more

Realm + Kotlin just got even better with Realm 4.3.0

by /

We’re excited to announce first class support for Kotlin in Realm Java 4.3.0! 🎉

If you’ve been following us closely, or seen us talk about Kotlin, then you already know that we’ve been busy at work, trying to streamline and simplify Realm Kotlin development.

In 3.6.0, we added additional nullability annotations to avoid leaking Java Platform Types and increase the overall null safety of our API.

Prior to 3.6.0

// Platform types were returned, so..
val result: String? = realm.where(Dog::class.java).findFirst() // was legal 
// -- or --
val result: String = realm.where(Dog::class.java).findFirst() // WAS ALSO LEGAL!!
// If no results found and null got returned, callers got an IllegalStateException

After 3.6.0

val result: String? = realm.where(Dog::class.java).findFirst() // is legal 
// -- but --
val result: String = realm.where(Dog::class.java).findFirst() // is illegal!!

Version 3.6.0 also took advantage of the Kotlin type system to infer required fields based on their types.

Prior to 3.6.0

//  An explicit annotation was necessary to indicate a required field, even though it was expressible in the Kotiln type system.
@Required 
var name: String = ""

After 3.6.0

// Implicitly @Required
var name: String = ""

// Implicitly not @Required
var name: String? = null

What’s new in 4.3.0!

We’ve added several new features for our growing Kotlin user base with this release, including:

Parameterized Functions

Using inline functions w/ reified type parameters, we were able to simplify our API so that when calling createObject, delete, where, you can just pass the type argument in <> and avoid passing the runtime class.

// Instead of this…
val dog = realm.createObject(Dog::class.java)  // create a Dog
realm.delete(Dog::class.java) // Delete All Dogs
realm.where(Dog::class.java)
     .findAllAsync()
     .asFlowable() // Fetch all dogs asynchronously as an RxFlowable

// You do this
val dog = realm.createObject<Dog>()  // create a Dog
realm.delete<Dog>() // Delete All Dogs
realm.where<Dog>()
     .findAllAsync()
     .asFlowable() // Fetch all dogs asynchronously as an RxFlowable

// Where Dog implements RealmModel or extends RealmObject.

New Extensions

We’ve added replacements for Realm and RealmObject static utility methods. Wherever your Java code called static Realm.xxxx() and RealmObject.xxxx(), those can be replaced with extension method calls.

The immediately apparent gain here is prettier looking code.

// Given a model class implemented using the model class interface instead of the base class
@RealmClass
open class Dog : RealmModel {
  var name: String = "" 
}

// Instead of 
RealmObject.addChangeListener(dog, changeListener)

// you can do 
dog.addChangeListener() // Because addChangeListener is added as an extension to anything that implements RealmModel.

However, a secondary win here is discoverability of API functions available. Whereas before, you may have had to look at the docs to understand that method RealmModel.xxxx was available, now Android Studio will give you content assist and auto-import the extension for you.

New Keyword

We’ve added the anyOf search term as a replacement for in.

In Java, in order to query for dogs matching any of a list of names, you might do something like this:

RealmResults dogs = realm
             .where(Dog.class)
             .in("name", new String[] {"Larry","Curly","Moe"})
             .findAll();

… but in, is a keyword in Kotlin. So the equivalent query in Kotlin looked like this:

val dogs = realm
     .where<Dog>()
     .`in`("name", arrayOf("Larry", "Curly", "Moe"))

… which just looked a bit awkward. So we’ve added a new anyOf keyword, you can use from Kotlin.

val dogs = realm
     .where<Dog>()
     .anyOf("age", arrayOf("Larry", "Curly", "Moe"))

Kotlin Model Generation

Finally, we are happy to announce that we’ve added schema model generation back into RealmStudio, our cross platform, next generation version of the Realm Object Browser. In addition to exporting (Swift, C#, JavaScript, and Java), we now also export to Kotlin! 🎉

Great, so how do I use all of this?

That’s the best part! To use all of this Kotlin specific functionality, all you have to do is use Realm with Kotlin! Since Realm gets included in your project as a Gradle Plugin, we can detect when you’re using Kotlin and enable all of this new functionality for you!

At Realm, we want to make persistence as easy as possible, so you can focus on what makes your app unique. Kotlin helps us in that mission.

Cheers!

The Realm Team

Read more

Realm World Tour Wraps for 2017 – See You Next Year!

by /

The Realm Platform Experience

2017 brought not one but two major platform updates. We kicked off the year with 1.0 release and went on tour visiting 40 cities across the globe just in time to launch Realm Platform 2.0.

RWT-banner

40 Cities
11 Realm Speakers
2000+ Attendees

RWT-photos

In case you missed us in a city near you, you can watch the replay!

Eric Maxwell provides a thorough introduction to the Realm Platform complete with code samples in Swift, Java and JavaScript as well as a live demonstration using the platform to build a 2-player realtime game, Realm Pop. This particular recording from our stop in NYC at iOSoho has a great technical Q&A throughout the presentation. If you’ve wondered how Realm might integrate with your backend systems, how it handles conflict resolution, or some key feature differences between our Developer, Professional, and Enterprise editions of the platform, you’ll definitely want to watch this one!

 

We’re gearing up for our 2018 Realm World Tour! Want us to come to your city? Tweet @Realm with hashtag #Realmworldtour and #cityname. Stay tuned to Realm tour news for our 2018 tour dates!

Read more

Announcing Realm .NET: Leveling Up Our Microsoft Support with Global Notifier & UWP Sync

by /

The Realm Platform helps you build cross-platform native applications, with sophisticated reactive features, realtime data sync, and robust offline experiences. It has two foundational components – the Realm database, an embedded client-side live object database, and the Realm Object Server, a realtime synchronization and event handling server – and these have worked great for anyone handy with Linux or Mac, Node.JS, and PostgreSQL. But, if you were a .NET stack developer, with a preference for C# and Windows, and maybe some important data in SQL Server, Realm only met you halfway. You could use Realm Xamarin to build native client-side apps with C#, and you could rely on our .NET Core support to start using Realm in server-side applications, but most of the rest of Realm required you to work outside of the standard Microsoft stack.

Until today. We’re pleased to announce Realm .NET, with robust support for critical Microsoft stack technology across the Realm Platform. We’re shipping three new things that are available today: the new Global Notifier for .NET, new sync on Windows/UWP, and data encryption support for Windows/UWP. In addition, we’re announcing a new SQL Server Connector, making two-way data integration between Realm and SQL Server dead simple, and giving mobile dev teams a much better way to bring legacy databases into their realtime, offline-ready applications. The SQL Server Connector is in private beta today, but will be available to Enterprise Edition customers beginning in December.

Demos speak louder than words, so we’ve created a simple demo video as a quick introduction. Then read on for a deep dive focused on the Global Notifier, and find more details on Realm .NET in our docs.

 

Technical Deep Dive

When we shipped .NET Core support in the summer, we enabled you to start using Realms in your server-side applications. This made it effortless to distribute new data to all your connected clients by just writing it to the user’s Realm on the server and letting Realm handle the synchronization for you. Reacting to changes by users wasn’t that easy though, as when you have thousands of user-Realms, any of which can change at any given time, keeping an open instance for each one of them is simply not feasible. This is where the Global Notifier steps in – it is heavily optimized to observe huge amounts of Realms and notify you whenever any of them changes.

Essentially, the Global Notifier is a NuGet package you add to your .NET project (it can be either .NET Core or a classic .NET Framework project, such as Asp.Net, console app, or even WPF app). Then, you implement the INotificationHandler interface that exposes two methods that are called by the SDK itself: ShouldHandle allows you to specify which Realm paths you’re interested in observing. HandleChangesAsync will be invoked when an observed Realm changes with detailed information about the state of the Realm just before and just after the change. Finally, you start the notifier by calling Notifier.StartAsync) and pass it all the notification handlers you want to register. And the cool part is, you can deploy this app anywhere you want – it doesn’t have to sit on the same instances that are running your Realm Object Server.

Let’s see how we could use it to create a handler that will analyze the sentiment of a support ticket by using Azure’s Text Analytics API. If you want to skip the step-by-step tutorial, jump right ahead to the completed sample.

Prerequisites

Before we start, you’ll need:

Setting Up the Project

Create a .NET Core app and add the following NuGet packages:

Setting Up the Notification Handler

The next step is to implement a notification handler. We’ll use the built-in RegexNotificationHandler as it provides an easy implementation of the ShouldHandle method in terms of Regex matching. We can handle all user realms at the /~/feedback path by using the ^/.*/feedback$ regex:

public class FeedbackHandler : RegexNotificationHandler
{
    private readonly SentimentClient _sentimentClient;
    public FeedbackHandler() : base($"^/.*/feedback$")
    {
        _sentimentClient = new SentimentClient(your-api-key)
        {
            Url = https://YOUR-REGION.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment”
        };
    }

    public override async Task HandleChangeAsync(IChangeDetails details)
    {
        // We’ll populate this in a bit
    }
}

We’ve created the basic skeleton and added the sentiment client, now we only need to start using it. To add some context, let’s assume that the objects the client is inserting and we’re handling have the following structure:

public class Ticket : RealmObject
{
    public string Description { get; set; }

    public float Score { get; set; }

    // Other irrelevant properties
}

Then, the first step toward implementing HandleChangeAsync will be to check if we have any insertions for that class:

if (!details.Changes.TryGetValue("Ticket", out var changeSetDetails) ||
    changeSetDetails.Insertions.Length == 0)
{
    // We don’t care about other objects being updated
    return;
}

Then, we can construct the SentimentRequest that we’ll send to Azure for analysis:

var request = new SentimentRequest
{
    Documents = changeSetDetails.Insertions
                                .Select((obj, index) => new SentimentDocument
                                {
                                    Id = index.ToString(),
                                    Text = obj.CurrentObject.Description,
                                    Language = "en"
                                })
                                .Cast<IDocument>()
                                .ToList()
};

What this is doing is sending the Description value of our Tickets as the text we’ll analyze and assigning the consecutive index of the object in the array as the Id of the document we’re analyzing. The document Id needs to be unique just for the request scope, so that approach is perfectly fine.

Next, we’ll send the request and process the response:

var response = await _sentimentClient.GetSentimentAsync(request);
var analyzedDocuments = response.Documents.Select(d =>
{
    var ticket = changeSetDetails.Insertions[int.Parse(d.Id)];
    return new
    {
        Score = d.Score,
        TicketReference = ThreadSafeReference.Create(ticket.CurrentObject)
    };
});

What’s interesting here is that we’re creating a ThreadSafeReference for the object. The reason is that, even though we’ll change it on the same thread, details.CurrentRealm, to which ticket.CurrentObject belongs, is read-only (as it is pinned to a specific version and we wouldn’t want things to change while processing the changes) and we’ll open a different Realm instance to write the changes to the ticket. To obtain the same object in a different instance of the same Realm, we can pass the object by its primary key or use ThreadSafeReference which works even for objects without a primary key.

Finally, we can write the score to the tickets:

using (var realm = details.GetRealmForWriting())
{
    foreach (var doc in analyzedDocuments)
    {
        var reference = realm.ResolveReference(doc.TicketReference);
        realm.Write(() => reference.Score = doc.Score);
    }
}

Starting Up the Notification Handler

To start up the notification handler we just created, we’ll need to pass it to a NotifierConfiguration. But before that, we’ll need to login an admin user:

SyncConfiguration.SetFeatureToken(YOUR-ROS-FEATURE-TOKEN);

var credentials = Credentials.UsernamePassword(username, password, createUser: false);
var admin = await User.LoginAsync(credentials, new Uri("http://127.0.0.1:9080"));

Next, we create the configuration:

var config = new NotifierConfiguration(admin)
{
    Handlers = { new FeedbackHandler() },
};

And finally, we start the notifier:

var notifier = await Notifier.StartAsync(config);

Make sure to keep a reference to the notifier for as long as you wish to receive notifications. When you want to stop your application, make sure to dispose of it to avoid any native memory leaks.

Seeing It in Action

To test it out, you can create a small console app that opens a regular synchronized Realm and adds Ticket objects. Something like:

var config = new SyncConfiguration(user, new Uri(http://127.0.0.1:9080/~/feedback”));
using (var realm = Realm.GetInstance(config))
{
    while (true)
    {
        var input = Console.ReadLine();
        If (input == exit)
        {
            break;
        }

        realm.Write(() => realm.Add(new Ticket
        {
            Description = input
        }));
    }
}

Completed Sample

You can find the complete sample, along with a user and employee apps in our Feedback Manager repo.

Read more

Bringing Native Performance to Electron

by /

The Native vs. Non-Native Dilemma

Realm initially supported the Apple ecosystem when we launched, so making the Realm Browser a native macOS app was a no-brainer. Today we offer broad cross-platform support for Android, Xamarin, React Native, and more. Given our new diversity, we needed to support a broader range of desktop platforms as well, but with limited resources we quickly realized we couldn’t build and maintain native apps for macOS, Windows, and Linux.

We knew Electron could give us the ability to write a cross-platform desktop solution with a single codebase, but we were reluctant to write our flagship user interface tool in a non-native framework. We were not sure if the performance of an Electron app would provide the right product impression of Realm.

It also didn’t make us feel super confident to see a ton of developers pile on the most visible Electron app for not being native enough:

So how did a company dedicated to creating better native cross-platform development find ourselves in facing this classic dilemma, and which path did we take?

Realm History: From Browser to Studio

Let’s take a step back for a moment to see where we came from. When Realm launched in 2014, Swift was still the new kid on the block, and at Realm, we were focused on supporting the Apple ecosystem with Objective-C and Swift APIs. Along with our new database, we introduced the Realm Browser, a native macOS application that offered a visual interface into Realm databases. We felt strongly that developers needed great tooling, and we chose to build a native application because it mirrored the philosophy behind Realm’s database design: pairing high performance with APIs designed for easy integration into native applications.

Fast-forward to 2017 and Realm has evolved tremendously. We support Android, Xamarin, React Native, including non-mobile platforms like Node and .Net. In addition, we launched the Realm Object Server, offering automatic realtime data sync.

We felt like it was time to build a better Realm Browser that reflected the breadth of Realm’s reach. Realm is now used by over 100,000 developers, but Realm Browser only worked on macOS, leaving Windows developers in the Android and Xamarin communities without the ability to visualize their data. Plus, the Realm Browser was written in Objective-C, and after all these years the codebase needed some refactoring. A new version would have to improve on a few things:

  • Cross-platform
  • High performance and modern codebase
  • Support for data browsing and editing of local and synced Realm databases
  • Support for administering Realm Object Server

Despite our concerns about Electron performance and our industry’s frustration with Slack, we believed that supporting our entire developer ecosystem was important. That, and there was so much love for Electron-based VSCode that we figured it might actually be possible to build a great desktop app in JavaScript. With that hope, Realm Studio was born.

realm-studio

We chose Typescript to ensure the codebase was easier to maintain, and we also decided we would combine the Realm Object Server’s web dashboard into the application. True to our roots, we also wanted to challenge ourselves to deliver native performance. This is the story of how we did it.

A Tale Of Two Processes

Architecturally, Electron is unique in that it is all built around Node, but consists of two (or more) processes—main and renderer(s). These processes are isolated and run concurrently to each other. This structure is based off Chromium which uses separate processes to isolate each tab or window so that a misbehaving webpage doesn’t affect others. The main process creates BrowserWindow instances. A BrowserWindow instance displays HTML via its own renderer process. Thus, if an Electron app is showing two windows, there would be three processes: one main process and two individual renderer processes.

electron processes

The main process controls the lifecycle of the entire application while the renderer process is mainly responsible for displaying its own content. These processes have minimal ability for communication and were designed this way for performance and stability reasons.

This immediately presents a challenge: how do these two independent and isolated process communicate with each other in a stable, performant way?

Electron offers several mechanisms, including APIs for interprocess communication: ipcRenderer and ipcMain. Both offer RPC style communication to pass messages between renderer processes and the main process. For a complex app, there will be a need to share state between processes which creates a synchronization problem. Slack recently blogged how their team adopted electron-redux which uses IPC to synchronize independent Redux stores.

electron-redux architecture

Instead, Realm offers a much more elegant solution to this problem. One of the lesser known features of Realm is that it supports interprocess access! Internally, Realm uses an Multi-Version Concurrency Control architecture to provide thread or process confinement. Multiple concurrent readers are allowed across threads and processes, while only a single writer is allowed. Realm’s architecture ensures each thread or process has a stable view and coordinates updates on writes, offering ACID compliance.

For Electron this is a perfect fit because now there can be a single store that is shared across the main and renderer processes!

Even better is that Realm offers reactivity, in which all changes to the Realm can be observed and objects or collections auto-update. This results in a single-data flow architecture, where for example, an update can be written to the Realm in the main process and then observed in a renderer process to automatically update the UI.

electron

Native Performance

By using Realm as the central store for data across the Electron processes, this opens up the question on whether accessing the Realm from the renderer process will cause any performance problems. The biggest complaint with web applications compared to native is that they tend to face a bigger hurdle in ensuring high frame rates when scrolling for example. If the data access is too slow between the redraw cycles than frames will be missed causing jerky behavior.

Realm’s design is focused on performance and for mobile applications we encourage accessing Realm directly on the UI thread. The reason Realm is capable of this is due to its zero-copy architecture and use of lazy loading. Realm utilizes memory mapping to provide high read performance, in addition, when performing queries or traversing an object graph, Realm does not have to load all the objects until each are accessed.

The net effect is that pairing Realm with a high performance UI framework, will result in no lag and low memory use for large collections. Opportunistically, there was already a great library for React called react virtualized that handles efficiently rendering large collections of data. Our initial testing confirmed that pairing Realm with react virtualized resulted in no lag in scroll performance for Realm files that had millions of rows!

As a result, we were able to achieve native performance with Realm Studio. See for yourself in the video below, or try it out on your own.

 

Going Forward

Looking back, we are really happy with Realm Studio and our use of Electron. Realm matches well to the architecture and we were able to take advantage of another great aspect of Electron, auto-updating, to deliver an even better experience than the original macOS browser.

We encourage you to try Realm out in an Electron application today! Just install the latest version of realm-js:

npm install realm

While Electron works great for our use in Realm Studio, we haven’t yet tested it out in other use cases and are therefore not fully ready to offer the same level of support as for our other supported products. We hope the community is interested in stepping in and help test and support this area going forward.

For example, while both the main and renderer processes use Node, there are some subtle distinctions between the two. Specifically, realm-js uses the libuv event loop to deliver notifications and in the renderer Electron combines the uv loop on top of the Chromium loop. Given our commitment to the Electron platform with Studio, we will be continuing to test these areas and we would love to hear your feedback on our forums)!

Read more