Cloud computing is changing the way IT services are accessed and consumed.  We are seeing that the dependence on infrastructure expertise is diminishing by engaging higher up the stack.

I described an end-to-end data pipeline in my first post.  I shall now show how to build the data capture and ingest processing as a flow in Node-RED purely through configuration without writing a line of code.  Node-RED offers flow-based programming for the Internet of Things and is available at nodered.org and on the IBM Cloud.

My flow implements a straightforward pattern.  Firstly, I have a node that reads data off an MQTT feed, then I undertake some data wrangling, which in this case lifts the json message payload to the top level of the document.  As we shall see in a subsequent post, this processing could be arbitrarily complex analytics and manipulation.  Finally, I write the documents into a Cloudant NoSQL database.

See how on YouTube.

The video shows how I am able to provision the Node-RED flow environment and the Cloudant database as one pre-configured service, ready for immediate use.

We shall see how we access the data I have captured for analysis in my third post.  In the meantime, try implementing the flow for yourself using the Node-RED Starter service on IBM Cloud.

This is the second in a series of posts on building an end-to-end data pipeline.  You can find my Node-RED flow and the other data pipeline artifacts on GitHub.