IoT data system design: Google Pub / Sub vs Kafka vs Kinesis vs PubNub to receive IoT data?

Question

I am trying to create an IoT + data analytics system, and I am having problems deciding which technologies or services should be used for swallowing.

Description of the ultimate goal at a high level:

• IoT devices enter data into the IoT gateway (using Zigbee, Z-wave, Bluetooth, etc.).
• The IoT gateway (which is connected to the Internet) pushes data to the pub / subsystem
• Base services process data coming out of the pub / subsystem, update dashboards and send alerts.

My question is which pubsub system should be used if we only need a response time of 10 seconds? (For example, the following is acceptable: the IoT device senses the event, and then after about 10 seconds it appears on the user panel or issues a warning)

Below are some questions:

• I see that PubNub advertised a lot for use in IoT. I understand that PubNub is essentially a very fast pubsub system that guarantees delivery in less than 1/4 of a second - is this the correct understanding of this? But PubNub is a bit more expensive compared to using Google Pub / Sub or supporting our own Kafka.
• Is Google Pub / Sub in a way similar to PubNub because, unlike a single Kafka self-managing cluster running in the same data center, Google Pub / Sub has its own network because it is part of Google? (similar to how PubNub is a "data stream network"?)
• If I use Kafka, should the manufacturer be in gateway devices?
  • If the manufacturer is not located in the gateway devices, should the Kafka manufacturers be on our servers and have a REST API to receive messages from the gateway devices?
  • If the manufacturer of Kafka IS is in the gateway devices, should there be anything special for Kafka brokers so that they can receive messages from the gateway devices?
• PubNub can be used to send commands back to IoT devices. Can this also be done using Google Pub / Sub or Kafka?
  • To output commands to IoT devices. With Kafka, does each gateway device need a consumer waiting for messages from those to which he has subscribed? (e.g. teams)

Also, not sure if it's worth mentioning, but at the moment the team is just me and maybe 2 more full stack developers. We read about Kafka and Zookeeper, but none of us passed a case study.

Answer 1

I would recommend option 3.1, as I personally know that it has been proven in production for many IoT uses, including one that includes more than 20 million devices. Confluent Kafka REST Proxy is open source and a great way to convert REST / HTTP from gateways over the Internet (using a firewall and load-balanced HTTPS protocol) to the Kafka REST proxy server in the cloud / data center, and then to Kafka and all the toolbar tools The back panels that support Kafka are very good. Even IBM uses this architecture for its IoT infrastructure on the Bluemix MessageHub. If you don't like REST, you can choose from MQTT, CoAP, websockset, AMQP, XMPP and many other Kafka connectors.

Answer 2

All great questions. (Full disclosure working for PubNub)

PubNub is more than just pub / sub (I will come back to this later). The first PubNub was built as a global distributed network for a reason - so we could provide low latency connectivity for all devices around the world through local points of presence. Because of this distributed architecture, your devices will always connect to the POP closest to them, and also because we replicate messages globally, if the server or host goes down, you will automatically connect to the next nearest host without losing messages. Because of this, PubNub provides 99.999% SLA for all users.

When it comes to the decision to “build against the purchase”, I can tell you that many of our customers started by thinking that they could build it themselves, but quickly realized that it was much more than they expected - more here . Creating and maintaining client libraries, scaling the server side, monitoring 24-7 and security are all that you need for your own experience. If you take into account the initial development costs, ongoing maintenance - instead of downloading the SDK today, start coding today to quickly enter the market with a well-known scalable solution.

PubNub's price is transaction-based, so depending on the number of devices and the level of traffic, I would argue that the total cost will still be less than one full-time employee. And for this you can choose one of 70+ client SDKs, use the proven scalable architecture, take advantage of the already built-in security features, prepare a complete team and wait for round-the-clock help so that you can focus on innovation, not infrastructure.

PubNub is more than just Pub / Sub. PubNub provides not only real-time messaging, but also stateless and serverless computing through their programmable network . PubNub allows you to write and deploy functions on the network, in fact there are already 30+ ready-made functions available in the BLOBKS PubNub directory that allow you to send SMS, emails and much more when changing data.

PubNub also created an open source project for creating real-time dashboards called Project EON . This makes it easy to visualize all the data in your device in real time.

You are also right in saying that PubNub can be used to remotely control a device . And today Insteon, Logitech, Samsung, Wink and many others are used precisely for this.

If you have further questions, PubNub has an excellent support staff available 24/7 - [e-mail] or click the chat button on the website.