AI/LLM-integrated Apps Penetration Testing
This is a beginner’s attempt to create a comprehensive collection of methodologies, learning materials, and other resources that are relevant in pentesting AI/LLM based or integrated apps.
When you apply for a credit card nowadays, there is a chance that you will solely interact with a bot or some form of AI. It may no longer necessary to make phone calls or send emails to agents or bank employees. A lot of web applications or mobile applications are built with AI/LLM integration.
The goal of this post is to have a personal repository of known methodologies and recommended learning resources for assessing the risks of AI/LLM-integrated Applications specially on the topic of Prompt Injection, which is number 1 in 2023 version of OWASP top 10 for LLMs. The focus is on assessing the apps that are integrated with AI/LLM and not primarily assessing the AI model itself.
Learning the Basics:
DeepLearning.AI’s “AI For Everyone” through Coursera
Google’s “Introduction to Generative AI” through Coursera
DeepLearning.AI’s “Generative AI with Large Language Models” through Coursera
Intel’s “AI Fundamentals Specialization” through Coursera
Impact of ChatGPT on YouTube
Learn Prompt Engineering from Freecodecamp
There are other courses for beginners from different universities on Coursera. There are courses also on Udemy and LinkedIn on Prompt Engineering.
Methodologies/Standards/Cheatsheets:
OWASP Top 10 for LLMs 2023
MITRE ATT&CK
Hacking LLMs with Prompt Injections
Art of Hacking LLM Apps
LLM Hacker’s Handbook
Recommended Courses:
Learn Prompting’s Introduction to Prompt Engineering and Advance Prompt Engineering
Learn Prompting’s Intro to Prompt Hacking and Advance Prompt Hacking
Portswigger Academy’s LLM Attacks
Coursera’s Introduction to Prompt Injection Vulnerabilities
Labs/Bugbounty Platform:
Huntr by Protect Ai
harishsg993010’s DamnVulnerableLLMProject
Damn Vulnerable LLM Agent
https://gandalf.lakera.ai/
https://gpa.43z.one/
https://github.com/ScottLogic/prompt-injection
Honorable mention
Below are some of the AI or GPT bots that are tailored for cybersecurity. You need to have a ChatGPT Plus subscription to use these bots. The section attempts to cover how we can use AI in pentesting or red teaming.
ATT&CK Mate by Cyb3rWard0g
https://chat.openai.com/g/g-fCIE7hCLx-att-ck-mate
HackTricksGPT by hacktricks.xyz
https://chat.openai.com/g/g-aaNx59p4q-hacktricksgpt
HackerGPT
https://www.hackergpt.co/
Resource on Adversarial Machine Learning
Offensive ML Playbook
https://wiki.offsecml.com/Welcome+to+the+Offensive+ML+Playbook
hxxps://www.atlan.digital/train/machine-learning-for-red-teams
Other Recommended Reads:
https://github.com/protectai/ai-exploits
https://owasp.org/www-project-ai-security-and-privacy-guide/
https://www.cobalt.io/blog/prompt-injection-attacks
https://atlas.mitre.org/techniques/AML.T0051
NIST’ Adversarial Machine Learning
https://learnprompting.org/docs/prompt_hacking/injection
https://academy.fuzzinglabs.com/ai_cybersecurity_hacking
https://github.com/f/awesome-chatgpt-prompts
By attempting to consolidate these resources, the post aims to provide a comprehensive guide for individuals delving into the realm of AI/LLM-integrated application penetration testing.
I am open to any feedback on the list above. This is new for me and I am still learning what is going on. Please feel free to add anything, make any constructive criticisms, or suggest any changes. I am eager to learn about the automated tools you use in your workflow.
