Cloud computing is changing the way IT services are accessed and consumed. We are seeing that the dependence on infrastructure expertise is diminishing by engaging higher up the stack.
I described an end-to-end data pipeline in my first post. I shall now show how to build the data capture and ingest processing as a flow in Node-RED purely through configuration without writing a line of code. Node-RED offers flow-based programming for the Internet of Things and is available at nodered.org and on the IBM Cloud.
My flow implements a straightforward pattern. Firstly, I have a node that reads data off an MQTT feed, then I undertake some data wrangling, which in this case lifts the json message payload to the top level of the document. As we shall see in a subsequent post, this processing could be arbitrarily complex analytics and manipulation. Finally, I write the documents into a Cloudant NoSQL database.
The video shows how I am able to provision the Node-RED flow environment and the Cloudant database as one pre-configured service, ready for immediate use.
We shall see how we access the data I have captured for analysis in my third post. In the meantime, try implementing the flow for yourself using the Node-RED Starter service on IBM Cloud.
This is the second in a series of posts on building an end-to-end data pipeline. You can find my Node-RED flow and the other data pipeline artifacts on GitHub.
Pingback: Simplifying data science | Tech Insights
Pingback: Building an end-to-end data pipeline | Tech Insights