Strategy for Success When Moving Machine Data Into The Cloud
In the past few months, I’ve been working with several customers, helping them move their machine data into the cloud. Each client is unique, and yet, I can’t help but see similarities between them. I wanted to share my observations as it is indicative of one of the most common use cases of Apache NiFi for IoT (Internet of Things).
From my observations, there are 3 key factors for successful digital transformation of your manufacturing business: company led initiative to move from being hardware manufacturer to create a correlated data offering, migration from legacy data transfer process to streaming data flows, and a project initiated by an engineering director with a cross-functional team of software developers, hardware engineers, systems architects and data engineers.
Augmenting manufacturing with data offerings
To begin with, I was struck by the interest in innovation by the leadership teams. All the companies I have engaged with have embarked upon a strategic initiative to leverage data to augment or enhance their business as hardware manufacturers. They want to extract more data from equipment deployed in customers’ environments to either enable engineers to build better products, help customers optimize processes utilizing their hardware or export data to enrich existing datasets for 3rd party insights.
Data they strive to collect is specific to their equipment, for example engine idle speeds, equipment uptime logs, error codes, voltages on custom hardware, patterns of usage, restarts per day, frequency of temperature changes, location information, power consumption, flow meter readings, valve open/close rates, pressure sensor readings and more. Although the equipment generates specific data sets that are unique to every company, the requirements are not. The need to capture and move data from the machine in the field, to create a new data offering is a common theme across companies.
Upgrading legacy methods of data transfer
The fundamental requirement to enable new data offerings is the ability to extract data from the equipment once it is sold and in use at customer sites. Typically extracting information is difficult and very limited. Deployed equipment often has very basic capabilities for transferring data due to its reliance on legacy capabilities of the hardware and the corresponding communications systems in place. For example, a common method of transferring data from equipment deployed in the field is to email a daily log file to an FTP server and then have an automated script email parse the data. This is a rigid, static and low frequency process that results in limited visibility of machine data.
Moving machine data in the cloud
There is a significant shift in interest in capturing and moving data in a more dynamic and streamlined process which is why customers are engaging with us. Apache NiFi offers an off the shelf option for moving the data off equipment deployed in the field to enable the new data offerings manufacturers are looking to create. It also enables migrating machine operations and performance data into the cloud for the visibility needed for a competitive edge in terms of asset performance management and operations efficiency.
Engineering led project
Beyond having similar strategic goals to grow data-based revenue streams and requirements for data movement, another commonality I have noticed is in the structure of the teams we engage with. The typical data movement project we have seen is lead by an engineering director tasked by management to find a way to create a new commercial offering based on the data that can be collected from deployed equipment.
These engineering leaders typically work with research and development teams of 6-8 people tasked with this transformational task. These teams span multiple disciplines including software development, hardware engineering, systems architecture, data engineering and/or data science. These teams study the business and the data to create project requirements alongside the creation of the custom code they need, essentially operating as a collapsed all-in-one team of business consulting and technical implementation to create a new data offering for the company.
Due to the nature of the specific equipment they manufacture each company’s needs are unique, and this small team needs to build a tailored solution. It is during this research and development phase where Calculated Systems gets involved. To achieve the goal of building a transformational data offering, the typical process consists of research for initial discovery and proof, followed by the creation of minimum viable product and then following up to provide concrete value on a reliable scale. To help accelerate this process we are now offering a 90-days to IoT program to help you get started on your path to moving your machine data into the cloud.
From my experience building IoT dataflows for multiple companies, I definitely believe that the 3 key factors for a successful digital transformation of your manufacturing business are: a strategic vision for data transformation, migration from legacy data transfer processes to streaming data flows, and a project initiated by an engineering director. If you can identify with any of these 3 criteria I would love to hear from you to compare notes and discuss how we can help you.
In the meantime, you may be interested in the following resources:
-
Learn more about Apache NiFi and IoT for Manufacturing
-
Read the case study “A New IoT Datastore” where we helped one firm capture data from over 25 different models of devices
-
Learn how Calculated Systems helped deliver a system that enabled detection of a malfunctioning machine in minutes instead of hours
-
Learn how we leveraged IoT dataflows to help a pharmaceutical manufacturing company reduce yield loss from 14.5% to 1.6%, without any capital investment or retooling
