IoT has rapidly moved from a fringe technology to a mainstream collection of techniques, protocols, and applications that better enable you to support and monitor a highly distributed, complex system. One of the most critical challenges to overcome is processing an ever-growing stream of analytics data, from IoT security data to business insights, coming from each device. Many protocols have been implemented for this, but could logs provide a powerful option for IoT data and IoT monitoring?
The incredible power of a network of IoT devices comes from the sheer volume and sophistication of the data you can gather. All of this data can be combined and analyzed to create actionable insights for your business and operations teams. This data is typically time-series data, meaning that snapshots of values are taken at time intervals. For example, temperature sensors will regularly broadcast updated temperatures with an associated timestamp. Another example might be the number of requests to non-standard ports when you’re concerned with IoT security. The challenge is, of course, how to transmit this much data to a central server, where it can be processed. IoT data collection typically produces a large volume of information for a centralized system to process.
The thing is, this is already a very well-understood problem in the world of log analytics. We typically have hundreds, if not thousands, of sources (virtual machines, microservices, operating system logs, databases, load balancers, and more) that are constantly broadcasting information. IoT software doesn’t pose any new challenges here! Conveniently, logs are almost always broadcast with an associated timestamp too. Rather than reinventing the wheel, you can simply use logs as your vehicle for transmitting your IoT data to a central location.
When your data is centralized, you can also begin to make predictions. For example, in the world of IoT security, you may wish to trigger alarms when certain access logs are detected on your IoT device because they may be the footprint of a malicious attack. In a business context, you may wish to conclude certain trends in your measurements, for example, if the temperature has begun to increase on a thermostat sharply, and its current trajectory means it’s going to exceed operational thresholds soon. It’s far better to tell the user before it happens than after it has already happened.
This is regularly done with log analytics and metrics. Rather than introducing highly complex and sophisticated time-series databases into your infrastructure, why not leverage the infrastructure you already have?
When you’re building out your complex IoT system, you’re inevitably going to need to build out your observability stack. With such a complex, distributed infrastructure, IoT monitoring and the insights it brings will be essential in keeping your system working.
This system will need to handle a high volume of traffic and will only increase when your logging system is faced with the unique challenges of IoT software. For example, logs indicate the success of a firmware rollout across thousands of devices worldwide. This is akin to having thousands of tiny servers that must be updated. Couple that with the regular operating logs that a single server can produce, which should put your IoT monitoring challenge into perspective.
Log analytics provide a tremendous amount of information and context that will help you get to the bottom of a problem and understand the overall health of your system. This is even more important when you consider that your system could span across multiple continents, with varying degrees of connectivity, and these devices may be moving around, being dropped, or worse! Without a robust IoT monitoring stack that can process the immense volumes associated with IoT data collection, you’re going to be left confused as to why a handful of your devices have just disappeared.
With this increased observability comes the power to detect and mitigate security threats immediately. Take the recent Log4Shell vulnerability. These types of vulnerabilities that exist in many places are challenging to track and mitigate. With a robust observability system optimized for the distributed world of IoT security, you will already have many of the tools you need to avoid these kinds of serious threats.
Your logs are also in place for as long as you like, with many long-term archiving options if you need them. This means that you can respond instantly, and you can reflect on your performance in the long term, giving you vital information to inspect and adapt your ways of working.
IoT security, observability, and operational success are a delicate balance to achieve, but what we’ve explored here is the potential for log analytics to take a much more central role than simply an aspect of your monitoring stack. A river of information, from your devices, can be analyzed by a wealth of open source and SaaS tools and provide you with actionable insights that can be the difference between success and failure.