Ingesting IoT data without writing code

Cloud computing is changing the way IT services are accessed and consumed.  We are seeing that the dependence on infrastructure expertise is diminishing by engaging higher up the stack.

I described an end-to-end data pipeline in my first post.  I shall now show how to build the data capture and ingest processing as a flow in Node-RED purely through configuration without writing a line of code.  Node-RED offers flow-based programming for the Internet of Things and is available at nodered.org and on the IBM Cloud.

ingest-flow

My flow implements a straightforward pattern.  Firstly, I have a node that reads data off an MQTT feed, then I undertake some data wrangling, which in this case lifts the json message payload to the top level of the document.  As we shall see in a subsequent post, this processing could be arbitrarily complex analytics and manipulation.  Finally, I write the documents into a Cloudant NoSQL database.

See how on YouTube.

The video shows how I am able to provision the Node-RED flow environment and the Cloudant database as one pre-configured service, ready for immediate use.

We shall see how we access the data I have captured for analysis in my third post.  In the meantime, try implementing the flow for yourself using the Node-RED Starter service on IBM Cloud.

This is the second in a series of posts on building an end-to-end data pipeline.  You can find my Node-RED flow and the other data pipeline artifacts on GitHub.

2 thoughts on “Ingesting IoT data without writing code

  1. Pingback: Simplifying data science | Tech Insights

  2. Pingback: Building an end-to-end data pipeline | Tech Insights

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s