Introduction

Welcome back to Neuvik’s Artificial Intelligence (AI) Risk Series – a blog series where we break down emerging cybersecurity trends related to AI. This blog provides detailed insight on common cybersecurity issues stemming from AI use.

At a glance:

  • Internal and external misuse can easily occur – and many organizations remain unaware of common insider threat use cases
  • Configuration and implementation challenges create significant risk – even for organizations building their own GenAI-integrated tooling in-house
  • Access controls often allow privilege escalation by default
  • Data privacy and confidentiality can easily be compromised, especially when users have not been trained on how to securely use GenAI-integrated tooling

Last time, we discussed what makes AI risk different, highlighting unique factors in the AI “tech stack” that can cause organizations to overlook sources of risk. In this blog, we’ll build on those learnings.

Common Cybersecurity Issues Stemming from AI

As we’ll discuss in subsequent blogs, AI has a unique attack surface that allows adversaries to target AI using novel techniques. However, before addressing the long tail of cybersecurity issues stemming from AI use, we recommend beginning with the “low hanging fruit” – i.e., the fundamentals that reduce the most likely sources of risk.

These include configuration and implementation challenges, privilege escalation by default, data privacy and confidentiality concerns, and opportunities for internal and external misuse.

Configuration and Implementation Challenges

As everyone working in cybersecurity knows, misconfigurations and insecure implementation are an adversary’s dream. AI presents an especially rich environment for configuration- and implementation-related exploits, as we discussed in our overview of the AI tech stack and the unique risks it creates. So, what can your organization do to prevent these cybersecurity issues stemming from AI use?

Perform diligence and verify controls promised by AI vendors. As with SaaS applications, perform diligence before on-boarding and do not inherently trust the commercial developer to implement appropriate security controls. In Neuvik’s research, many commercial AI-integrated tools suffer from a lack of security logging, insufficient jailbreak protections, and limited content filtering (i.e., insufficient limitations when a user asks certain questions about certain subjects). Similarly, use pre-purchase risk questionnaires to assess the application’s security posture and review permissions that the application has access to or utilizes to perform key functionality.

When building applications in-house, be sure to consider how desired use cases interact with the underlying AI system – are you building something that requires it to perform functionality it wasn’t designed or tested for (and, if so, what risks does that create)?

Implement security logging and ensure logs integrate with security event management and orchestration tools. Visibility is critical. Adversaries can leverage AI for data exfiltration and as a jumping off point for lateral movement. Many organizations also overlook the very real possibility that insiders may purposefully or inadvertently misuse AI tools. Configuring security logging can provide critical visibility into user behavior and can provide a more in-depth situational awareness for event analysis.

Implement jailbreak protections and content filtering. Jailbreak protections refer to security measures that prevent users from bypassing restrictions intended to restrict the AI’s use to a set of specific use cases. These restrictions aim to prevent AI from answering harmful, unethical, or unauthorized questions. On the flipside, content filtering helps to block harmful, sensitive, or inappropriate content from being accepted as input. Configuring both controls can significantly reduce risk but often go overlooked.

Privilege Escalation by Default

As with most enterprise applications, AI-integrated applications often require access to sensitive file systems and databases. Further, they often require account management capabilities and may utilize multiple APIs to interface with both internal and external resources. This all sounds standard – until you realize that, yet again, basic security hygiene is often overlooked.

These access requirements introduce risk in two ways:

  • Access for these tools typically enables privilege escalation by default
  • Accounts used to access AI tools often fall outside traditional Identity and Access Management programs (especially in the case of “shadow AI” unmanaged by the enterprise), allowing users to essentially “bypass” role-based permissions

How could this play out in reality? Let’s take an employee – Sally – who works in Human Resources. Sally uses an account management bot to process HR requests and interface with a cloud-based service that hosts her company’s employee benefits portal.

Sally normally can’t access her peers’ or manager’s salary or personal data from her regular account. However, the HR account management bot needs access to this data to function correctly. The HR bot also uses an API to connect to the benefits portal – and has access to the API keys. Sally discovers that, while logged into the HR bot, she can convince it to escalate her permissions using basic manipulation techniques – and, in an attempt to be “helpful” the bot also provides her the API keys. Sally has now achieved privilege escalation and, if desired, could exfiltrate sensitive employee data or move laterally to the employee benefits portal.

Let’s illustrate the second risk. Consider an employee – Joe – who would like to modify data in a database as part of his workflow. When logging into the database via his own account, Joe can only view and export snippets of the database. However, Joe’s team recently introduced a helper bot. The helper bot’s account has full administrative access to the database (likely due to a misconfiguration or default permissions that weren’t changed during implementation!). Crafty Joe realizes he can just use the GenAI bot to make the changes – essentially bypassing the appropriate role-based permissions he “should” have had on his account.

Not only does this example showcase risks from inappropriate access controls, but it shows how oversight of any of these fundamental best practices can compound into a major vulnerability.

“Oversight of any of these fundamental best practices can compound into a major vulnerability.”

To mitigate these risks, ensure appropriate role-based access controls and restrict access to specific AI tools to only those with a “need to know” rationale.

Data Privacy and Confidentiality Concerns

Data privacy is perhaps the most well-known concern when it comes to AI security.

These concerns typically fall into one of two buckets:

  • Risk that confidential or non-public information could be inadvertently incorporated into the “corpus” (or training data) of GenAI tooling
  • Worries that GenAI-integrated tooling can be manipulated to disclose sensitive data, allowing malicious actors (or insiders!) to gain access to or exfiltrate this information

Concern about the inclusion of confidential or non-public information in the “corpus” of AI tooling isn’t without merit – in fact, it’s already happened. In 2023, Samsung discovered an employee had uploaded IP into ChatGPT, leading the company to ban its use. In another example, a major research institution in the USA disclosed to Neuvik that they’d discovered scientific teams uploading pre-publication data into GenAI-integrated tools, using it to enhance their analysis. These scientists were unaware that their data – and findings – could be exposed publicly as a result, an especially dire risk in a “publish or perish” research context.

We recommend:

  • Increasing user awareness and providing targeted AI risk briefings
  • Providing “sandboxes” or local instances for employee use
  • Building GenAI-integrated tooling in-house, trained on a purpose-built dataset

Fortunately, these tips can reduce the risk of incorporating confidential or non-public information significantly.

Opportunities for Internal and External Misuse

Of course, even with the correct configurations, appropriate access controls, and data privacy controls, it’s possible for both internal and external misuse to create risk.

Insider Threat. There are three primary types of internal misuse perpetuated by either negligent or malicious insiders, beyond bypassing access controls. These include “model poisoning” during the Corpus Generation process, incorporation of “bad artifacts” into work products, and the use of computer vision to exfiltrate large amounts of data.

Corpus Generation refers to the process of training AI, and a key insider risk is the ability to train the AI to consider risky activity as “normal” and effectively “poison” the baseline. A malicious insider could also train AI to return specific results, influencing workflows or creating dependencies on poisoned data.

To make this risk concrete using a cybersecurity example, a malicious insider could tune the “flow rate” of a Data Loss Prevention (DLP) tool such that it treats a significant data flow as “normal” – when, in reality, this should suggest an exfiltration event. Similarly, most Endpoint Detection and Response (EDR) tools use a baselining component. If a malicious actor ran malware on the endpoint while training the baseline, it would accept the “bad” behavior as normal. In both cases, the baseline has been “poisoned” due to insider behavior during the corpus generation process.

The next area of risk from insider misuse is the introduction of “bad artifacts” into work product. Often, this results from directly copying from a GenAI-integrated tool without human oversight. These “bad artifacts” can range widely. They can include the introduction of insecure code or the creation of dependencies on outdated or vulnerable code libraries. They can also include the incorporation “hallucinations” into work product. Hallucinations are created when the underlying GenAI service doesn’t have sufficient information to answer the prompt and generates a fictitious, often nonfactual response. Fortunately, “bad artifacts” can often be identified with human oversight or best practices such as automated code testing.

Lastly, insiders can introduce risk via misuse by utilizing computer vision or Optical Character Recognition (OCR) to completely bypass security controls. OCR can be used to quickly parse and recreate data sets on an external system. Effectively, this means an employee could use their personal phone to exfiltrate and recreate data from their professional computer. Most often, this technique supports corporate espionage, the sale of intellectual property, or various other nefarious goals. Fortunately, security software can test for indicators of this risk, such as the rapid opening of hundreds of files in succession.

External Misuse. External actors can also misuse AI-integrated tooling – most often by manipulating the underlying AI system or bypassing the security configuration and access controls mentioned earlier in this blog.

AI’s complex attack surface and susceptibility to unique attack types presents an appealing target for adversaries. These include:

  • Evasion attacks, in which adversaries subtly modify inputs to trick the AI into making ‘dangerous’ decisions
  • Prompt injection, which manipulates AI by taking advantage of it’s desire to be “helpful” and its ability to be flattered or intimidated in a human-like manner
  • Model poisoning – as with insiders, external actors who have access to the AI system can alter its functionality to poison results or spread mis- or dis-information
  • Context manipulation and memory injection, which essentially aim to manipulate the AI into modifying the underlying “guardrails” or “context” (i.e., the logic of the model)
  • Overwhelm of human-in-the-loop, which operates similarly to a Denial-of-Service attack – essentially, attackers can identify which inflection points receive human intervention and target them, essentially overwhelming human operators with alerts and diverting attention or taking systems offline

Often, these attacks take advantage of AI’s non-deterministic behavior – in other words, its propensity to provide a different result when faced with the same prompt. Attackers also have the benefit of time – and the ability to use AI itself to accelerate their malicious behavior.

Want to learn more? Our next blog explores this unique attack surface in detail.

Conclusion

Common cybersecurity issues stemming from AI look similar to those in traditional technology but are vastly more complex due to AI’s attack surface and functionality. Taking the steps suggested in this blog can reduce risk from these common issues.

Check back next week for a deep dive into what makes AI’s attack surface unique and a detailed explanation of unique attack types targeting AI.

Ready to learn more about managing AI risk at your organization? Check out the latest AI Risk Management service offerings and contact us to schedule a consultation today.  

Interested in learning more about insider threat? Hear it directly from Neuvik’s Director of Cyber Risk Management, Celina Stewart, in a collaboration with NextLabs: AI & Insider Threat Part 1 and AI & Insider Threat Part 2.