Introduction

Welcome back to Neuvik’s Artificial Intelligence (AI) Risk Series – a blog series where we break down emerging cybersecurity trends related to AI. For our final blog in the series, we explore gaps in AI risk management frameworks, including:

  • The NIST AI 100-1 and 600-1 Risk Management Frameworks (“NIST AI RMF”) highlight the importance of data privacy, yet they do not impose specific requirements regarding encryption standards, access control measures, or differential privacy techniques for safeguarding AI-related data
  • The NIST AI RMF emphasizes the need for ethical development of AI technology – however, they fail to provide guidance for the ethnical deployment of AI by organizations primarily purchasing or licensing AI tools
  • Existing frameworks do not define an AI-specific security testing protocol, which may lead organizations to overlook critical vulnerabilities in their AI systems
  • AI Risk Management Frameworks do not provide tactical guidance on changes to user awareness training and acceptable use needed to address AI risks

Gaps in AI Risk Management Frameworks

Artificial Intelligence (AI) is transforming industries, revolutionizing daily life, and expanding the limits of technological capabilities. Yet, as AI systems become increasingly advanced and autonomous, are organizations considering the growing inevitability of risks associated with AI? Issues such as security vulnerabilities, potential manipulation, and existential threats highlight the importance of thoughtful assessment and regulation of AI. However, current AI risk frameworks fail to tackle these challenges, leaving AI systems vulnerable to security threats.

Neuvik identified the following gaps when conducting extensive research on the existing AI Risk Management Frameworks and accompanying risk management guidance. This research looked specifically at NIST AI 100-1, NIST AI 600-1 (together referred to as the NIST AI Risk Management Framework), the OECD “AI Principles” and ethics considerations proposed by UNESCO and IBM. While each of these frameworks presents fantastic guidance for organizations building and implementing Generative Artificial Intelligence (GenAI) and Generative Adversarial Networks (GAN) tooling, we did identify several gaps.

Insufficient guidance on security for high-risk AI systems

Some AI technologies, including those used for facial recognition or the creation of “deepfakes” and audio cloning, create more significant risks than others. While the organizations utilizing these tools do so for legitimate use cases (for example, authentication via Face ID), the technology underlying them remain ripe for abuse.

Existing AI risk management frameworks outline broad principles for managing risks associated with GenAI and GAN technologies (especially during development) but do not offer targeted guidelines for organizations deploying these high-risk AI applications. Further, the frameworks lack guidance on how to prioritize among types of AI applications to ensure those deemed “high risk” or “critical” receive stricter controls.

Lack of controls to prevent AI-specific adversarial attacks

AI models, particularly those leveraging machine learning to adjust their outputs for a given user, are especially vulnerable manipulation-based attacks. Often, these manipulation techniques include appeals to AI’s human-like desire to “get along” and “help” and its propensity to become “distracted” or “intimidated” if competing or forceful requests occur.

For example, GenAI’s human-like desire to “please” can lead to some chatbots succumbing to flattery and praise. Worryingly, a crafty adversary can not only use this manipulation to expose sensitive data but actually convince the AI that they should have access to restricted data by pretending to have a legitimate reason.

Beyond manipulation, AI tools are also vulnerable to a unique set of attacks targeting the way they function. Check out our prior blog, “Unpacking AI’s Unique Attack Surface and Attack Types” to learn more.

While the NIST AI RMF emphasizes the importance of robustness in security controls, it does not provide comprehensive technical guidance for organizations on how to defend against these specific threats.

Further, because most of the AI RMF guidance relates to the development of AI technology, companies who license or purchase pre-built AI tools may not realize the underlying models themselves could be vulnerable, even if their organization applies application-layer controls. The lack of emphasis on “deployment” control is a major gap in AI risk management frameworks.

Insufficient emphasis on AI supply chain and vendor security

AI systems frequently depend on external models, open-source libraries, and cloud services, increasing the risk of “supply chain” attacks. For organizations licensing or purchasing their AI tools, it may not be intuitive to treat AI vendors as “suppliers” – leaving those organizations open to risks.

AI supply chain and vendor management threats include:

  • Unknown or unwanted data embedded in pre-trained models, such as malicious code or intellectual property from other organizations
  • Compromised data sources (such as vulnerable open-source code or outdated libraries) utilized for training
  • Security vulnerabilities in APIs and cloud environments used to host / run AI technology
  • Assumptions that AI vendors provide adequate security controls, such as access controls, encryption, data privacy, etc.

Fortunately, the NIST AI RMF does provide guidance on how to map AI technology and legal risks when dealing with third-party data or software. Some vendor management and supply chain provisions also exist. However, the existing framework lacks definitive best practices for securing AI supply chains, such as tracking model provenance, verifying dependencies, and conducting cryptographic integrity checks.

Weak guidance on data privacy requirements and controls

AI systems depend on extensive datasets, which frequently include sensitive or confidential personal or professional data. This data can include customer data (PII), healthcare data (including patient data or ePHI), trade secrets or intellectual property, and unstructured data such as Human Resources files or database extracts.  

Given the magnitude of data used, AI models are an incredibly attractive target for adversaries. Although the NIST AI RMF emphasizes the significance of privacy (especially when considering inputs and training data), it does not mandate specific requirements for encryption standards, access control protocols, or differential privacy methods to protect data related to AI.

Specifically, Neuvik found that the AI RMF did not provide specific requirements for:

  • Data loss prevention / protection
  • Access controls
  • Encryption
  • Configuration management
  • Logging / monitoring, including integration with Security Incident & Event Management tools

The lack of specific recommendations for these cybersecurity controls is especially concerning for tools built on AI providers like ChatGPT, as organizations may not recognize ChatGPT as a “vendor” in their AI supply chain and may assume that controls already exist. We discuss these risks within the AI “tech stack” in depth here: https://neuvik.com/article/what-makes-ai-risk-different/ .

No standardized recommendation for security testing of AI systems

AI models require unique a unique and multi-layered security testing approach, setting them apart from traditional software. While the NIST AI RMF does provide guidance on how to perform security testing for models in development, it falls short in providing directive recommendations for organizations licensing or purchasing their AI tools.

Unfortunately, the NIST AI RMF lacks provisions for both offensive and defensive security testing for AI systems. On the offensive side, the NIST AI RMF does not explicitly recommend for testing that would mimic actual cyber threats, such as AI-specific penetration testing or Red Teaming, nor does it require automated AI vulnerability assessments. On the defensive side, the NIST AI RMF does not explicitly require periodic updates to ensure model algorithms have not drifted or changed, nor does it recommend checking to ensure baselines have been trained appropriately. This could leave models at risk from external actors and insider threats.

In the absence of standardized, AI-specific security testing, organizations risk missing significant vulnerabilities within their AI systems.

Limited focus on AI model theft and intellectual property protection

AI models developed “in-house” can be valuable intellectual property (IP), yet they are vulnerable to theft or duplication through methods such as model extraction attacks. These attacks enable malicious actors to repeatedly query AI models to understand and replicate their capabilities, to take advantage of API weaknesses to gain access to sensitive model architectures, or to pilfer proprietary models from cloud storage or endpoints.

Unfortunately, the current NIST AI RMF framework does not provide targeted strategies to safeguard AI models against these attacks. To address this gap, Neuvik recommends restricting unauthorized access and using available protections like API security, rate limiting, or watermarking techniques to decrease the likelihood of a successful model extraction attack.

How can your organization protect itself?

The gaps have been identified, and now you might be wondering:

  • “Did we take these risks into account before integrating AI systems into our operations?”
  • “If the frameworks are lacking, how can I guarantee our compliance?”
  • “What steps can I take to assess the safety of our data?”

To answer these questions, we recommend:  

  • Avoid becoming complacent – use the NIST AI RMF and other frameworks as a starting point, but ensure cybersecurity best practices are in place as well
  • Keep an inventory of “high risk” or “critical” AI systems and enable appropriate additional controls for those systems
  • Perform AI Penetration Testing to identify susceptibility to AI-specific attacks
  • Implement an AI vendor management program
  • Utilize data privacy controls and seek visibility into data used by AI systems
  • Implement IP protection, where possible
  • Use both offensive and defensive security testing to inform risk

Not sure where to begin or looking for a partner in your AI Risk Management program? Let Neuvik help! We have deep expertise in AI Risk Management and AI Penetration Testing, with services desired to secure your organization – no matter what phase of your AI journey.

Ready to learn more? Contact us today or learn more about our services: https://neuvik.com/our-services/cyber-risk-management/.