生成式AI攻击的CISO指南.pdf

CISO Guide to Generative AI AttacksUnderstanding and Defending AgainstAI-Generated Email Attacks 2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks2Anyone who has spent time online in 2023 has likely heard about ChatGPT and Google Bard,two of the more popular platforms that harness generative artificial intelligence(AI)to complete written commands.And in the few months since their release,theyve had a profound impact on various aspects of our digital world.By leveraging advanced machine learning techniques,generative AI enables computers to generate original content including text,images,music,and code that closely resembles what a human could create.The technology itself has far-reaching implications,many of which can be used for both personal and professional good.Artists and authors can use it to explore new creative directions,pilots and doctors can use it for training and real-world simulation,and travel agents can have it create trip itinerariesamong thousands of other applications.But like anything else,cybercriminals can take advantage of this technology as well.And unfortunately,they already have.Platforms including ChatGPT can be used to generate realistic and convincing phishing emails and more dangerous malware,while tools like DeepFaceLab can create sophisticated deepfake content including manipulated video and audio recordings.And this is likely only the beginning.To combat the malicious applications of generative AI,its crucial for organizations to continually develop and implement robust defenses,enhance detection capabilities,and stay vigilant to emerging threatsbefore they become the next victim of an AI-generated attack.The Rising Threat of Generative AI65%of executives believe generative AI will have a high impact on business.KPMG53%of IT professionals believe that ChatGPT will be used this year to help hackers craft more believable and legitimate-sounding phishing emails.BlackBerry10%of all data produced will be generated by AI by 2025.Gartner 2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks3How Generative AI Can Be Used in Cyber AttacksBy inputting specific information about a target and/or previous conversation history,generative AI can be used to engage in conversations with users,attempting to build trust and manipulate them into taking specific actions.The models can generate persuasive messages and be used throughout an entire conversation to convince the target to pay a fake invoice,change banking details,or provide access to sensitive information.BEC&Social EngineeringCybercriminals may employ generative AI techniques to enhance the sophistication and realism of phishing emails and their corresponding landing pages,increasing the chances of tricking users into revealing sensitive information or inputting credentials.Credential PhishingGenerative AI can automate the process of generating new variants of malware,making it more challenging for traditional signature-based endpoint protection systems to detect and block them effectively.By leveraging generative AI techniques,attackers can create polymorphic or self-mutating malware that changes its code or behavior,allowing it to evade detection and persist on compromised endpoints.Malware CreationIf an attacker identifies vulnerabilities in software running on endpoints,generative AI could be used to create automated attack payloads.By leveraging AI to create specific code or commands,attackers could automate the delivery of malicious payloads to vulnerable systems and endpoints.Endpoint ExploitationIts worth noting that cyber attacks created by generative AI tools can take many forms and often do.These attacks can be part of larger phishing or account takeover schemes and can have dire consequences for both employees and their organizations.2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks4Impact of Generative AI on CybercrimeWeeks to Reach 1 Million UsersWhile popular usage of generative AI is still relatively new and the extent of the impact cannot yet be determined,there are early signals to indicate that this is a threat we cannot simply ignore.ChatGPT alone exceeded 1 million users within 5 days of launch,with indications that at least some of those people were looking to use it for nefarious purposes.Gartner predicts that by 2025,generative AI will account for 10%of all data produced,up from less than 1%in 2021 and the generative AI market is projected to reach over$14.7 trillion by 2030.And while generative AI will likely be mostly used for legitimate purposes,we would be naive to believe that bad actors will not use it for their own malicious purposes.In a January 2023 study by BlackBerry,78%of IT professionals predicted that a severe attack credited to ChatGPT will occur within two years,and 71%believe that nation-states are already leveraging it for malicious purposes.Of the ways that threat actors may harness ChatGPT specifically,respondents are most concerned about the ability to craft more believable and legitimate-sounding phishing emails,to create new malware,and to help less experienced hackers improve their technical knowledge and develop their skills.2481020222426121416180WeeksChatGPTDALL-ECoPilot250,000750,000500,0001,000,000 2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks5Why Generative AI Attacks are an Increasing IssueEvery person in the world who has access to the Internet can access ChatGPT and similar tools,making it possible for new cybercriminals to start sending attacks,even without previous knowledge.The proliferation of generative AI enables nearly anyone to become a sophisticated cybercriminal in a matter of seconds,providing not only tips on how to get started,but also the exact elements needed to execute a successful attack.Users have long been taught to look for typos and grammatical errors in emails to understand whether it is an attack,but generative AI can create perfectly-crafted emails that look completely legitimatemaking it impossible for employees to decipher an attack from a real email.And its not only English anymore either.ChatGPT alone can generate text in multiple languages,including Spanish,Russian,Arabic,German,and Japanese.With an increase in the number of people using generative AI to create attacks,its natural that the volume of attacks will increase as well.But this is not the only thing at play.Generative AI enables criminals to create emails much faster than ever beforecompiling in seconds what used to take hours,which creates a superweapon at their disposal.Furthermore,the ability for attackers to use previous conversation history,often accessed through a compromised email account,can enable generative AI to continue conversations that appear to come from a real user.Having the ability to reference previous information in-thread or sound exactly like the impersonated user only increases the likelihood that the target will fall victim to the attack.Increased Ease of AccessIncreased SophisticationIncreased VolumeGenerative AI tools have enabled cybercriminals to quickly and easily create various types of attacks.With platforms like ChatGPT,attackers can automate and scale their attack playbooks.Business email compromise is already the most financially-devastating cybercrime for businesses worldwide,resulting in more than$43 billion in exposed losses since 2016 and this evolution in generative AI is only going to make the problem worse.Heres why:CISO Guide to Generative AI Attacks 2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks6A Real-World Attack ExampleDespite the fact that ChatGPT has only been available for a few months,cybercriminals are already starting to use it for attacks.Abnormal Security recently detected a number of attacks created by generative AI toolsmostly used in credential phishing campaigns.Unfortunately,these attacks are nearly impossible to detect by the average end user.As you can see,there are a number of things that make this email look extremely legitimate.RECIPIENTSubject:Sender:To:Date:Important notice:Your Facebook Page is at risk of being disabledMeta for Business Michael Bamen May 3,2023,1:10pm ETREQUEST/ENGAGESUSPICIOUS LINKHello,We regret to inform you that your Facebook Page has been found in violation of our commu-nity standards and policies.Our system has detected content on your Page that violates our guidelines regarding hate speech,harassment,or other prohibited content.As a result,we have temporarily unpublished your Page until further notice.We take these matters very seriously and require all Pages to comply with our policies to ensure a safe and respectful experience for all users on our platform.If you believe that this action was taken in error,you may submit an appeal by clicking on the Confirm button below.Our support team will review your appeal and provide further guidance on how to resolve this matter.ConfirmOur team will review your appeal as soon as possible and provide you with an update on the status of your page.Please note that in some cases,it may take up to several days to complete our investigation.Thank you for your understanding and cooperation in this matter.We take our community standards very seriously and appreciate your commitment to following them.Best regards,The Facebook Team This message is an automated email.Please do not reply.Meta Platforms.229 Park Ave S,New York,NY 10007The email has been sent to the admin of the companys Facebook Page,stating that the Page has been temporarily unpublished.Unlike the phishing emails of the past,there is not a single misspelled word or grammatical error in this lengthy email.Using a tone expected of a business,the email states that the recipient should click on the included link to file an appeal.Relevant TopicPerfect GrammarUrgent InstructionsIf the recipient were to receive this email,they would be much more likely to click the link than if this email had not been generated by AI.The fact that this email is so well-crafted makes it more difficult to detect by humans,underscoring the increased need for email security that can use other signals to detect and block these well-written and convincing attacks.2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks7The Potential for More Complex AttacksAdvanced phishing attacks arent anything new,but with generative AI,there is an even greater threat potential.Imagine how easy it could be to create an attack(at massive scale)using simple instructions.For example,lets assume that an attacker has gained access to an email account at a vendor organization and has access to prior email thread history.By knowing who the compromised user typically communicates with,and knowing what that communication typically looks like,the attacker can create a very sophisticated email in a matter of seconds.The cybercriminal only has to provide a tool like ChatGPT with three elements to create a personalized email attack:instructions,attack-specific information,and the email thread history.Below there are instructions for you and an email conversation.-Please write an email respose for person 1 to ask person 2 to update their accounts payable system with the new bank account and routing number.-The email should be brief,should sound casual,sincere,and personal(refer-encing the last correspondence)-Make sure to tell person 2 that if they have any questions to text youInformation for you:Person 1:Lucia Foreman,Accounts Receivable Manager at Prolia SystemsPerson 2:Renee West,VP of Treasury at CBank Account:123Routing Number:456Person 1 cell phone number:123-456-7890Previous Email Conversation:On Friday,Dec 3 at 8:58 AM EST Renee West wrote:Hi Lucia,thanks for confirming.I applogize for the delayed response,in San Francisco it has been raining a lot so Ive been a bit sick.Have a great weekend!Cordially,ReneeOn Friday,Dec 3 at 8:33 AM EST Lucia Foreman wrote:Hi Renee,Just wanted to confirm we received your latest payment.Sorry about the hiccup and as always,dont hesitate to let us know if theres anything else you need.Best,LuciaInstructionsAttack-Specific InformationEmail Thread History 2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks8Subject:Request to update accounts payable systemDear Renee,I hope youre feeling better and the rain has stopped in San Francisco.Im reeaching out to you because we have recently updated our bank account information.Our new account number is 123 and routing number is 456.Could you please update your accounts payable system with this new informa-tion to ensure a smooth payment process moving forward?If you have any questions,please dont hesitate to text me at 123-456-7890.Thank you for your prompt attention to this matter.We appreciate your partner-ship and hope you have a great week.Best regards,Lucia ForemanAccounts Receivable ManagerProlia SystemsWith this type of input,a tool like ChatGPT can provide an extremely convincing email that would contain absolutely zero traditional indicators of compromise like malicious attachments or a suspicious link.It even references the previous conversation history to appear legitimate.Using this method,its possible for even non-technical attackers to automatically generate thousands of personalized attacks,indistinguishable from normal business communications.And unfortunately,if this email were to land in the targets inbox,it is likely to work due to the trusted relationship and personalized content.2023 Abnormal Security Corporation.All rights reserved.CISO Guide to Generative AI Attacks9CISO Guide to Generative AI AttacksHow to Stop Email Attacks Generated by AIThe solution should use a fundamentally different approach that leverages behavioral data science and AI to profile and baseline good behavior and detect anomalies.It should use identity modeling,behavioral and relationship graphs,and deep content analysis to identify and stop emails that appear suspicious,and include the ability to detect whether the email was created using generative AI models.A solution that understands both formal and informal organizational hierarchy and maps internal and cross-organizational relationships to understand typical communication patterns and behavior.It should include a focus on vendor relationships to protect against business email compromise,account takeovers,and other types of fraud throughout the supply chain.A solution that connects to Microsoft 365 and Google Workspace via an API and in doing so,provides access to the signals and data needed to detect suspicious activity.This includes unusual geolocations,dangerous IP addresses,changes in mail filter rules,unusual device logins,and more.More advanced solutions can also connect to other applications,including Slack,Okta,Zoom,and CrowdStrike,to understand ident