The Dequeuer: Push data into the Realm Object Server using a Message Queue
The aim of this exercise is to create a basic test application that follows the dequeuer architecture pattern for the Realm Platform. Upon completion, you will be able to write a message to Apache Kafka, consume the message via node.js, and respond to this consumed message by creating a new object in the Realm Object Server.
The exercises below require a running instance of the Professional or Enterprise edition of the Realm Object Server. Detailed installation instructions can be found here.
(Note: if you run into any issues during installation, make sure you have all of the required prerequisites).
You also need to create an admin user on your Realm Object Server. This username and password will be used in step 4.
Step 1: Download Kafka
Download the 0.11.0.1 release and un-tar it
[email protected]:~$ wget http://apache.mirrors.pair.com/kafka/0.11.0.1/kafka_2.11-0.11.0.1.tgz [email protected]:~$ tar -xzf kafka_2.11-0.11.0.1.tgz [email protected]:~$ cd kafka_2.11-0.11.0.1
Step 2: Start the Server
The following section requires java to be installed. This can be done via the terminal with:
[email protected]:~/kafka_2.11-0.11.0.1$ sudo apt-get update [email protected]:~/kafka_2.11-0.11.0.1$ sudo apt install default-jre -y
Kafka uses ZooKeeper, so you first need to start a ZooKeeper server. Kafka includes a script to get a quick single-node ZooKeeper instance running.
[email protected]:~/kafka_2.11-0.11.0.1$ bin/zookeeper-server-start.sh config/zookeeper.properties &
Now, start the Kafka server:
[email protected]:~/kafka_2.11-0.11.0.1$ bin/kafka-server-start.sh config/server.properties &
Step 3: Create a topic
In a new terminal window, create a Kafka topic
(Note: you can change the topic name to whatever you desire. I have chosen “kafkaTest”)
[email protected]:~/kafka_2.11-0.11.0.1$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkaTest
You can print out all of your existing topics via the following command:
[email protected]:~/kafka_2.11-0.11.0.1$ bin/kafka-topics.sh --list --zookeeper localhost:2181
Step 4: Create a consumer that ingests Kafka and writes to ROS
(Note: the consumer and producer that we create in the following steps will need to be saved and ran locally from the Kafka server)
Next, we need to create a Kafka consumer to take messages from the Kafka queue and write them to the ROS. We are going to do this via node.js
This relies on realm-js and no-kafka npm packages being installed in the working directory. You will need to install npm:
[email protected]:~/kafka_2.11-0.11.0.1$ sudo apt install npm -y
Now we can install the dependencies for the consumer via npm. You will need a certain version of node for realm-js which can be installed via these dependencies
[email protected]:~/kafka_2.11-0.11.0.1$ npm install realm [email protected]:~/kafka_2.11-0.11.0.1$ npm install no-kafka
You can find the code for our consumer here.
Within this code, you will need to edit a few things such as the ROS Address, your login credentials for Realm, and the name of your Kafka topic:
//Params to edit var ROS_Address = 'INSERT_ROS_ADDRESS_HERE' //i.e.[ ‘](http://localhost:9080/)[http://localhost:9080](http://localhost:9080)’ var username = 'INSERT_USERNAME_HERE'; //i.e. ‘realm-admin’ var password = 'INSERT_PASSWORD_HERE'; //i.e. ‘’ var kafkaTopic = 'INSERT_TOPIC_HERE'; //i.e. kafkaTest const token = "INSERT_FEATURE_TOKEN_HERE";
Once you’ve done this, save the code as
consumer.js to a convenient location on your Kafka server.
[email protected]:~/kafka_2.11-0.11.0.1$ node consumer.js
Step 5: Create a producer to send messages to Kafka
We first need to create a producer where we can generate messages for Kafka
You can do this via the terminal window with the following command:
[email protected]:~/kafka_2.11-0.11.0.1$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafkaTest
You can find the code for our producer here.
Note: This relies on no-kafka being installed in the working directory. If you are using the same directory as your consumer, this dependency was already installed.
The script will produce messages to kafka. You can change the content of these messages by changing the messageBuffer variable.
Once you are happy with the script, save it as producer.js on your Kafka server. It can then be run via the terminal with:
[email protected]:~/kafka_2.11-0.11.0.1$ node producer.js
Step 6: View the results via Realm Studio
Download and installation instructions for Realm Studio can be found here.