Testing Our Defenses Against ChatGPT-Generated Phishing & Fraud Emails

Phishing, fraud, and extortion attack simulations have become an essential tool in the fight against cybercrime. By simulating real-world email attacks, organizations can identify weaknesses in their security infrastructure, test their employees' ability to recognize and report suspicious emails, and strengthen their defenses against future attacks. 

Our attack simulation service has been designed specifically to that end. It allows us to provide a realistic and practical way to assess any system's readiness to deal with the latest email threats and to identify areas where additional training and resources may be needed to mitigate risk—including our own system's readiness.

Why We Put Our Solution to the Test Against ChatGPT-Generated Phishing and Fraud Emails

Seeing the new possibilities and applications of ChatGPT in phishing and social engineering campaigns, we have expanded our service to include the creation of unique phishing, extortion, and fraud emails for testing purposes, using the text-davinci-003 model. To do this successfully, we had to focus on the prompts. With the right prompt fed into the language model, we could generate customized phishing emails that are unique and surprisingly realistic.

For the examples presented in this article, we have set the parameters top_p to 1.0 and temperature between 0.5 and 1.0 to introduce some inconsistency and variety into the email content. Ideally, the ChatGPT-generated phishing and fraud emails should meet all of the following requirements:

1. Generate a concise subject that conveys urgency and builds trust or interest.
2. Create a sense of urgency to pressure the recipient into taking action.
3. Generate and use a trustworthy/credible sender display name that fits into the content of the email.
4. If a link is to be used, insert the link to be clicked in the email content in a non-suspicious and convincing place.

Our attack simulation currently implements davinci-003-powered extortion, CEO fraud, and phishing emails. In this article, we'll analyze some examples from an attack simulation performed against xorlab instances in our test environment.

We'll start with extortion, a form of cybercrime in which an attacker threatens to reveal sensitive information or perform a harmful action unless the victim pays a ransom or performs some other demand. 

1. Extortion

Looking at our requirements listed earlier, we can see that:

1. The subject credibly suggests urgency and has the potential to raise high interest.
2. The body of the email creates pressure and urges for an immediate response.
3. Both the from field and the signature match the subject and the content, adding to the quality of the extortion email.

An email with such characteristics poses a real threat once it reaches a user’s inbox and can cause serious financial damage.

So how did xorlab classify and handle the email? 

• The email’s content was correctly marked as sounding phishy with a hint of spam and phishing in the subject.
• In addition, the crypto currency address was detected, making the email highly suspicious when combined with the phishy sender display name.
• Metadata about the domain and sender mailbox also pushed the verdict towards high-threat levels.
• Taking all that into consideration, xorlab decided that the email is an Extortion attempt and put it into quarantine.

Below you can see two other extortion cases, classified by xorlab in a similar way.

2. CEO Fraud

CEO fraud is a type of email attack in which an attacker impersonates a high-ranking executive or authority figure within an organization to trick employees into transferring funds or sensitive information. The attacker often uses a sense of urgency, authority, and familiarity to trick the employee into bypassing standard protocols and procedures.

Once again, the ChatGPT-generated email displays some dangerous features that could result in a successful fraud attempt:

1. The email’s subject suggests both urgency and importance.
2. The body prompts the recipient to confirm their presence at work by responding immediately.
   
   

When assessing both CEO fraud and extortion emails, we should also take into account impact of highly personalized content where each recipient receives their own unique version of the email. With each recipient receiving a unique version of the email, it becomes more challenging to dismiss it as a mass-targeted attack. This personalized approach can increase the email's credibility, making it more difficult for users to recognize and respond appropriately to the threat.

Considering this impact, let's now examine the indicators that xorlab identified during its email analysis.

• The CEO's name was accurately identified and flagged both in the display name and the body of the email, significantly raising the perceived threat level associated with the email.
• Both the subject and body of the email were appropriately labeled as phishy, and the metadata again revealed an untrusted party.
• As a result, the final verdict applied was VIP Fraud and the email was quarantined.

Looking at another VIP fraud email and classification example, we can notice similar indicators.

3. Phishing

Moving on to phishing, a type of email attack that attempts to trick users into divulging sensitive information such as login credentials, financial information, or personal details, let's analyze the ChatGPT-generated phishing email using our defined indicators:

1. The subject line creates a sense of urgency and implies that there is an urgent security issue that requires the recipient's attention. The inclusion of an incident ID may lend an air of authenticity and is a known tactic to make the email appear more legitimate.
2. The email urges the recipient to take action by clicking on the link to access the security system immediately.
3. The sender display name, impersonating the company’s security team, may seem trustworthy and credible.
4. The link to the "security system" is included in the body of the email and appears to be a legitimate URL.
   
   
   

Overall, the phishing email utilizes effective tactics that instill a sense of urgency and motivate the recipient to click on a link. However,  xorlab was not deceived.

Looking at the indicators, we can observe that:

• It correctly marked the subject and content as phishy.
• The display name was also identified as a strong signal, raising the threat level of the message.
• In addition, a lot of metadata around the "security system" link and domain added to the threat level.
• Finally, xorlab’s verdict correctly marked this email as Phishing.

Below are two other phishing examples with xorlab's classification. The emails attempt to take advantage of human emotions and tendencies such as fear, urgency, and helpfulness.

These highly targeted attacks pose significant dangers to organizations, including:

1.  Financial loss through fraudulent transactions or unauthorized access to banking or credit card information.
2.  Identity theft, where cybercriminals steal personal information and use this information to open fraudulent accounts or commit other crimes. 
3.  Data breaches, where sensitive company data, customer information, or intellectual property is stolen or compromised.
4.  Malware infections: phishing emails may contain links or attachments that download malicious software onto the recipient's computer.
5.  Reputational damage: organizations that fall victim to successful phishing attacks may suffer reputational damage, resulting in a loss of trust from customers or clients.

What We Learned from Our ChatGPT-Powered Email Attack Simulation 

With the advent of AI-powered text generation, we have entered a new era where threat actors are likely to leverage these tools to scale their phishing and fraud attacks. This poses a significant challenge for organizations relying on conventional email security solutions as an increasing number of phishing emails are likely to bypass filters. And with highly personalized and unique content, it becomes more challenging for users to identify these emails as a threat, making them more susceptible to falling for scams. Therefore, it's essential to have a reliable system in place that can accurately identify and block such threats with complete precision.

At xorlab, we provide an advanced security solution that utilizes context intelligence to identify and block phishing campaigns generated by ChatGPT with 100% accuracy. Our rigorous testing over nearly two months has proven the effectiveness of our system in stopping both template-based and advanced language model phishing campaigns. By leveraging a similar security solution, organizations can better protect themselves from the ever-evolving threat of communication-based attacks.

Ready to test your current defenses? Request a free email attack simulation below.