Current Security Testing Platforms: Verification Methods and More

Table of Links

Abstract and 1 Introduction

2. Current Security Testing Platforms

2.1. Recent progress

3. A New Testing Platform and 3.1. Testing platform roles

3.2. Web-based remote access

3.3. Testbed setup

4. Enabled Testing Methodologies

4.1. Secure Development Lifecycle (SDL) testing and 4.2. Penetration testing

4.3. Research testing

5. Conclusion & Outlook, and References

2. Current Security Testing Platforms

Currently, we see two main categories of verification activities for vehicle cybersecurity that testing platforms have aimed to support. As a part of automotive engineering development activities, cybersecurity verification is done as a part of a standard V&V process. The usual procedure for aligning this with ISO/SAE 21434 Clause 10.4.2 is as follows:

1. Verification methods: This type of testing usually follow the ASPICE engineering process [17] of Unit Testing, Integration Testing, and Qualification Testing. Cybersecurity requirements are combined with functional requirements and verified as per engineering process standards. This type of testing includes requirement-based testing, interface testing, resource evaluation, control and data flow verification, and static and dynamic analysis. The techniques used in this type of testing follow general software testing methodology and can be easily performed by a qualified test engineer.

2. Testing for unidentified weakness or vulnerabilities: This type of testing is unique to the field of cybersecurity and requires the test engineer to have specialized knowledge of or experience with cyber attacks (e.g., the Sam Curry attacks [2]) in order to be effective. These activities include targeted functional testing, vulnerability assessment, fuzz testing, and penetration testing.

Out of the numerous cybersecurity testing activities discussed above, the verification testing methods are mostly addressed with a comprehensive engineering process. These methods stand to benefit from broader acceptance of other automotive quality-oriented disciplines, such as ASPICE and functional safety, which should lead to improvements. As the industry adopts better and more effective software quality engineering, in part, thanks to regulatory pressure, we are starting to see many tools being developed to address the activities of vulnerability assessment and fuzz testing, specifically targeting automotive use cases.

Challenges. Even with the entrance of more service providers into the field, automotive cybersecurity functional testing and penetration testing, particularly for complex components or systems, still suffer from shortcomings in technical proficiency and cost effectiveness. Below, we list some of the main challenges we see in these types of testing:

1. Technically qualified automotive cybersecurity testing specialists are hard to find,

2. Testing engagements are difficult to manage due to the (implicit) need to access hardware remotely, and

3. Fully remote penetration testing is still not realistic for most automotive use cases and difficult to repeat over time.

As evident in other cybersecurity disciplines, such as web security or Internet-of-Things (IoT) security, a test engineer should readily have access to targets. However, getting access to a test target is currently much more difficult for automotive pentesters due to the cost, availability, complexity of modern vehicle architectures, and equipment needed for car hacking. Due to the unique skill set of pentesters and the small talent pool, most automotive OEMs and suppliers do not have the internal capability to perform fuzzing or penetration testing on every product. Penetration testing requires a completely different skill set and mindset from the traditional test engineer or development engineer. This activity is increasingly outsourced to third-party service providers. Compared to internal V&V, this outsourcing is much more difficult to manage in the event of project delays, dependency on third-party expertise and resources, concerns about data security and confidentiality, and limited hardware availability, which can result in timeline penalties and potentially cause mistakes in test target configurations.

There has been some effort in addressing these issues with running software simulations of ECUs and networks on the cloud. However, these are often extremely limited and do not represent even a fraction of all attack surfaces of a modern vehicle [12]. More importantly, any hardwarebased weaknesses and vulnerabilities are often completely absent in software-only simulations. Traditionally, test beds for security projects are custom built for penetration testing with a highly customized interface. These test beds have a short lifetime (i.e., the length of a penetration testing contract), are very time consuming to build, and require specialized knowledge to interface with them. Due to the custom nature of these test beds and their fast turnarounds, it has not been worthwhile to build a generalized testing platform. As vehicles evolve to become more software-defined, a high demand is placed on testing far beyond product development and into the production and operation stages of the vehicle life cycle. Without a test bed to rerun penetration tests on, identifying new vulnerabilities during the life of a product can be difficult.