Faster, More Accurate IoT Security: A Quantitative Analysis of the CUMAD Framework

Table of Links

Abstract and 1. Introduction

2. Related Work

3. Background on Autoencoder and SPRT and 3.1. Autoencoder

3.2. Sequential Probability Ratio Test

4. Design of CUMAD and 4.1. Network Model

4.2. CUMAD: Cumulative Anomaly Detection

5. Evaluation Studies and 5.1. Dataset, Features, and CUMAD System Setup

5.2. Performance Results

6. Conclusions and References

5. Evaluation Studies

In this section we perform evaluation studies to investigate the performance of CUMAD using the publicdomain N-BaIoT dataset [8]. In order to better understand the evaluation studies, we will first describe the dataset, in particular, the features of the data points contained in the dataset. We will also compare the performance of CUMAD with that of the N-BaIoT scheme (which is the name for both the dataset and the corresponding scheme on detecting compromised IoT devices) [8].

5.1. Dataset, Features, and CUMAD System Setup

N-BaIoT contains both benign and (Mirai and Bashlite) attack traffic of 9 commercial IoT devices, including two doorbells (Danmini and Ennio), an Ecobee thermostat, three baby monitors (different models from Provision and Philips), two SimpleHome security cameras, and a Samsung webcam. Benign IoT device traffic was collected immediately after the corresponding IoT device was connected to the experimental testbed. Care was taken to ensure that various representative normal operations and behaviors of IoT devices were collected into the benign dataset.

In the N-BaIoT dataset, each data point corresponds to an arrived packet, and contains 115 statistical features, which together represent a behavioral snapshot that describes the context of the corresponding packet when it arrives at the data collection point. The snapshot contains the source and destination device information, the protocol information, among others. More specifically, the 115 features were extracted in the following manner. For each arriving packet, a total of 23 features were collected at different levels of aggregation (see Table 1), including features aggregated at source IP address level, at source MAC and IP addresses level, at level of channel (source and destination IP addresses), and at socket level (source and destination IP addresses and port numbers). These 23 features were extracted in a sliding window fashion, over 5 time windows of 100ms, 500ms, 1.5sec, 10sec, and 1min, respectively, generating a total of 115 features for each data point.

We use the Keras sequential model as the foundation for our development of the autoencoder [12]. The model’s input dimension is set to match the number of features in the dataset (that is, 115). To ensure effective compression, we implement three hidden layers within the encoder. These layers progressively reduce the dimensions to 87, 58, 38, and 29, respectively, with the last one (29) being the dimension of the output layer of the encoder, that is, the dimension of the obtained code. Conversely, the decoder component mirrors the dimensions of the encoder layers in the reverse order, starting from 38. By employing compression and decompression in the encoder and decoder layers, we effectively eliminate redundant information from the features of the input data points. To optimize training performance, we utilize the Adam optimizer, and the mean square error is used as the reconstruction error (objective function of the model).

SPRT requires four user-defined parameters in order to compute the upper and lower bounds A and B (see Eq. (3)), as well as the step function for computing Λn following each observation (see Eq. (1)). The desired values for both the false positive rate and the false negative rate (represented by α and β, respectively) are typically very small. In this study we set both α and β to 0.01. Ideally, the parameter θ indicates the true probability of an observation being classified as an anomaly, from either a benign or compromised IoT device. We determine the values for θ0 and θ1 through our preliminary studies, and set them to 0.2 and 0.8, respectively.

5.2. Performance Results

Table 2 shows the performance of CUMAD in detecting IoT devices, in terms of accuracy, recall, and F1 score [12]. From the table we can see that CUMAD achieves superior performance in all three metrics. For example, for 5 of the IoT devices, CUMAD is able to detect all the compromised cases (see the column of Recall). CUMAD is also able to detect vast majority of the compromised cases for the remaining two of the IoT devices, with recall scores of 0.999 and 0.994. Considering both detection precision of attack and benign traffic, we can see that CUMAD also performs very well, with an accuracy score ranging from 0.955 to 0.995 for all 7 IoT devices. The F1 scores, which is a weighted average of the precision and recall scores of a model, also confirm that CUMAD performs well in detecting compromised cases.

Figure 4 shows the false positive rates of an autoencoder based anomaly detection scheme and CUMAD. As shown in the figure, the false positive rates of the autoencoder-based anomaly detection scheme for the 7 IoT devices range from 0.77% to 11.22%, while the false positive rates of CUMAD range from 0.014% to 2.067%. On average the autoencoder based anomaly detection scheme has about 3.57% false positive rate, while the false positive rate of CUMAD is about 0.5%, which represents about 7 times performance improvement in terms of false positive rate for CUMAD over the autoencoder-based anomaly detection scheme.

For performance comparison, we also include in the table the performance results of the N-BaIoT scheme, with the same evaluation studies setup. We can see from the table that CUMAD and N-BaIoT performs comparably in terms of all three-performance metrics. However, N-BaIoT works on a fixed window size. Table 2 shows that N-BaIoT requires a relatively large window size, ranging from 20 to 82 (column with name Window Size). In contrast, CUMAD works in an online fashion and does not requires such a fixed window size. Table 2 shows the average number of observations required for CUMAD to reach a detection (column with name Mean Size); we can see from the table that it takes on average less than 5 observations for CUMAD to make a detection of a compromised case, much quicker than NBaIoT. In order to have a better understanding of the number of observations for CUMAD to make a detection of a compromised case, Figure 5 shows the cumulative distribution function (CDF) of required observations for CUMAD to make a detection for all the 7 IoT devices. We can see from the figure that the vast majority of detection requires less than 10 observations for all 7 IoT devices.

In summary, compared to simple anomaly detection schemes such as the ones only based on autoencoders, CUMAD can greatly reduce the false positive rates, making CUMAD much more attractive than simple anomaly detection schemes in the real-world deployment. Compared with window-based schemes such as N-BaIoT, CUMAD requires much less observations to reach a detection, and thus can detect compromised IoT devices much quicker.

6. Conclusions

In this paper we have developed CUMAD, a cumulative anomaly detection framework for detecting compromised IoT devices. CUMAD employs an unsupervised neural network autoencoder to classify whether an individual input data point is anomalous or normal. CUMAD also incorporates a statistical tool sequential probability ratio test (SPRT) to accumulate sufficient evidence to detect if an IoT device is compromised, instead of directly relying on individual anomalous input data points. CUMAD can greatly improve the performance in detecting compromised IoT devices in terms of false positive rate compared to the methods only relying on individual anomalous input data points. In addition, as a sequential method, CUMAD can quickly detect compromised IoT devices. Evaluation studies based on public-domain IoT dataset N-BaIoT confirmed the superior performance of CUMAD.

References

[1] Yin Minn Pa Pa, Shogo Suzuki, Katsunari Yoshioka, Tsutomu Matsumoto, Takahiro Kasama, and Christian Rossow. Iotpot: analysing the rise of iot compromises. In Proceedings of the 9th USENIX Conference on Offensive Technologies, pages 9–9, 2015.

[2] Michael Fagan, Katerina Megas, Karen Scarfone, and Matthew Smith. Foundational cybersecurity activities for iot device manufacturers. Technical report, National Institute of Standards and Technology, May 2020.

[3] Mohammed Ali Al-Garadi, Amr Mohamed, Abdulla Khalid Al-Ali, Xiaojiang Du, Ihsan Ali, and Mohsen Guizani. A survey of machine and deep learning methods for internet of things (iot) security. IEEE Communications Surveys & Tutorials, 22(3):1646–1685, 2020.

[4] Andrew A Cook, Goksel Mısırlı, and Zhong Fan. Anomaly detection ¨ for iot time-series data: A survey. IEEE Internet of Things Journal, 7(7):6481–6494, 2019.

[5] Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):1–58, 2009.

[6] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.

[7] Abraham Wald. Sequential Analysis. John Wiley & Sons, Inc, 1947.

[8] Yair Meidan, Michael Bohadana, Yael Mathov, Yisroel Mirsky, Asaf Shabtai, Dominik Breitenbacher, and Yuval Elovici. Nbaiot—network-based detection of iot botnet attacks using deep autoencoders. IEEE Pervasive Computing, 17(3):12–22, 2018.

[9] Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1–38, 2021.

[10] Erol Gelenbe and Mert Nakıp. Traffic based sequential learning during botnet attacks to identify compromised iot devices. IEEE Access, 10:126536–126549, 2022.

[11] Thien Duc Nguyen, Samuel Marchal, Markus Miettinen, Hossein Fereidooni, N Asokan, and Ahmad-Reza Sadeghi. D¨ıot: A federated self-learning anomaly detection system for iot. In 2019 IEEE 39th International conference on distributed computing systems (ICDCS), pages 756–767. IEEE, 2019.

[12] Francois Chollet. Deep learning with Python. Simon and Schuster, 2021.