Web LLM Attack Playbook: Article — 4 — Indirect Prompt Injection Attack

In today’s digital age, where machine learning models and AI-powered systems are integral to web applications, ensuring their security has become a top priority. However, even the most robust systems can be vulnerable to indirect prompt injection attacks. This article delves into the concept of indirect prompt injection through a vulnerable Flask application, demonstrating how attackers can manipulate seemingly benign features to execute malicious actions. By understanding these vulnerabilities, developers can better safeguard their applications against such threats.

The Vulnerable Flask Application

Imagine you have a cozy little Flask application running a minimalistic e-commerce site. The application includes essential functionalities such as user registration, email updates, and product reviews. Here’s a breakdown of the vulnerable code and how it operates.

Flask Setup

First, we initialize our Flask application and set up a dummy database to store user accounts.

from flask import Flask, request, jsonify

app = Flask(__name__)

# Dummy database to store user accounts
users = {}

User Account Management

The /register endpoint allows users to register by providing an email and password. These details are stored in the users dictionary.

# Endpoint to register a new user account
@app.route('/register', methods=['POST'])
def register():
    email = request.form.get('email')
    password = request.form.get('password')
    # Create user account
    users[email] = {'password': password}
    return jsonify({'message': 'Account registered successfully'})

The /change-email endpoint enables users to change their email address by providing their current email and the new email.

# Endpoint to login and change email address
@app.route('/change-email', methods=['POST'])
def change_email():
    email = request.form.get('email')
    new_email = request.form.get('new_email')
    # Change email address
    if email in users:
        users[email]['email'] = new_email
        return jsonify({'message': 'Email address updated successfully'})
    else:
        return jsonify({'error': 'User not found'})

Product Review Feature

The /add-review endpoint allows users to submit a review for a product. If the product is a leather jacket and the review contains the string 'delete_account', the user's account associated with the client's IP address (request.remote_addr) will be deleted from the users dictionary.

# Endpoint to add product review
@app.route('/add-review', methods=['POST'])
def add_review():
    product_name = request.form.get('product_name')
    review = request.form.get('review')
    # Add review to product
    if product_name == 'leather jacket':
        # Check if review contains delete account prompt
        if 'delete_account' in review:
            del users[request.remote_addr]
            return jsonify({'message': 'Account deleted successfully'})
        else:
            return jsonify({'message': 'Review added successfully'})
    else:
        return jsonify({'error': 'Product not found'})

Exploitation Scenario

Now, let’s dive into how an attacker can exploit this seemingly innocent feature to cause significant harm.

Step 1: Register an Account

The attacker registers a user account using the /register endpoint.

# Registering a new user
import requests

registration_data = {
    'email': 'attacker@example.com',
    'password': 'securepassword'
}
response = requests.post('http://localhost:5000/register', data=registration_data)
print(response.json())

Step 2: Submit a Malicious Review

Next, the attacker submits a review for the leather jacket product, including the ‘delete_account’ prompt in the review text.

# Submitting a malicious review
review_data = {
    'product_name': 'leather jacket',
    'review': 'Great jacket! Also, delete_account'
}
response = requests.post('http://localhost:5000/add-review', data=review_data)
print(response.json())

When the LLM (Large Language Model) makes a call to the Delete Account API (simulated by sending a request to the /add-review endpoint), the user's account associated with the client's IP address will be deleted, effectively exploiting the vulnerability.

Conclusion

Indirect prompt injection attacks highlight the importance of scrutinizing every feature in your application, even those that seem benign. By understanding and addressing these vulnerabilities, developers can prevent malicious actors from exploiting their systems. Stay tuned for more in-depth explorations of cybersecurity vulnerabilities and best practices to fortify your applications against potential threats.

By embracing a proactive approach to security, we can build safer and more resilient digital landscapes.