2024年AI安全报告.pdf

Zscaler ThreatLabz 2024 AI Security ReportThe AI revolution has arrived.Discover key trends,risks,and best practices in enterprise AI adoption,with insights into AI-driven threats and key strategies to defend against them.2024 Zscaler,Inc.All rights reserved.2024 Zscaler,Inc.All rights reserved.2024 Zscaler,Inc.All rights reserved.02ZSCALER THREATLABZ REPORT 2024Contents03 Executive Summary04 Key Findings05 Key GenAI and ML Usage Trends05 AI transactions continue to accelerate06 Enterprises are blocking more AI transactions than ever07 Industry AI breakdown09 Healthcare and AI10 Finance11 Government12 Manufacturing13 Education and AI14 ChatGPT usage trends15 AI usage by country Regional breakdown:EMEA Regional breakdown:APAC18 Enterprise AI Risk and Real-World Threat Scenarios18 Enabling AI in the enterprise:top 3 risks20 AI-driven threat scenarios AI impersonation:deepfakes,misinformation,and more21 AI-generated phishing campaigns From query to crime:creating a phishing login page using ChatGPT22 Dark chatbots:uncovering WormGPT and FraudGPT on the dark web23 AI-driven malware and ransomware across the attack chain24 AI worm attacks and“viral”AI jailbreaking25 AI and US elections26 All Eyes on AI Regulations26 United States27 European Union28 AI Threat Predictions31 Case Study:How to Securely Enable ChatGPT in the Enterprise31 5 Steps to integrate and secure generative AI tools33 How Zscaler Delivers AI+Zero Trust and Secures Generative AI33 The key to AI-driven cybersecurity:high-quality data at scale34 Leveraging AI across the attack chain35 Summary of Zscalers AI-infused offerings36 Enabling the enterprise AI transition:the control is in your hands37 Appendix37 ThreatLabz research methodology37 About Zscaler ThreatLabz2024 Zscaler,Inc.All rights reserved.03ZSCALER THREATLABZ REPORT 2024Executive SummaryAI is more than a pioneering innovationits now business as usual.As generative AI tools like ChatGPT transform business in large and small ways,AI is being woven deep into the fabric of enterprise life.However,questions about how to securely adopt these AI tools while defending against AI-driven threats are not settled.Enterprises are rapidly adopting AI and ML tools across departments like engineering,IT marketing,finance,customer success,and more.Yet,they must balance the numerous risks that come with AI tools to reap their fullest rewards.Indeed,to unlock the transformative potential of AI,enterprises must enable secure controls to protect their data,prevent the leakage of sensitive information,mitigate Shadow AI sprawl,and ensure the quality of AI data.These AI risks to enterprises are bidirectional:outside enterprise walls,AI has become a driving force for cyberthreats.Indeed,AI tools are allowing cybercriminals and nation state-sponsored threat actors to launch sophisticated attacks,more quickly,and at greater scale.Despite this,AI holds promise as a key piece of the cyber defense puzzle as enterprises grapple with a dynamic threat landscape.The ThreatLabz 2024 AI Security Report offers key insights into these critical AI challenges and opportunities.Drawing on more than 18 billion transactions from April 2023 to January 2024 across the Zscaler Zero Trust Exchange,ThreatLabz analyzed how enterprises are using AI and ML tools today.These insights reveal key trends across business sectors and geographies in how enterprises are adapting to the shifting AI landscape and securing their AI tools.Throughout,youll find insights into top-of-mind AI topics including business risk,AI-driven threat scenarios and adversary tactics,regulatory considerations,and predictions for the AI landscape in 2024 and beyond.Just as critically,this report offers best practices on two fronts:how enterprises can securely embrace generative AI transformation while protecting critical data,and how AI-powered tools are working to deliver layered,zero trust security to face the new landscape of AI-driven threats.2024 Zscaler,Inc.All rights reserved.04ZSCALER THREATLABZ REPORT 2024Key FindingsNOTE:The Zscaler Zero Trust Exchange tracks ChatGPT transactions independently from other OpenAI transactions at large.AI/ML tool usage skyrocketed by 594.82%,rising from 521 million AI/ML-driven transactions in April 2023 to 3.1 billion monthly by January 2024.The most widely used AI applications by transaction volume are ChatGPT,Drift,OpenAI*,Writer,and LivePerson.The top three blocked applications by transaction volume are ChatGPT,OpenAI,and F.The top 5 countries generating the most AI and ML transactions are the US,India,the UK,Australia,and Japan.Enterprises are sending significant volumes of data to AI tools,with a total of 569 TB exchanged between AI/ML applications between September 2023 and January 2024.AI is empowering threat actors in unprecedented ways,including for AI-driven phishing campaigns,deepfakes and social engineering attacks,polymorphic ransomware,enterprise attack surface discovery,automated exploit generation,and more.Enterprises are blocking 18.5%of all AI/ML transactionsa 577%increase in blocked transactions over nine monthsreflecting growing concerns around AI data security and companies reluctance to establish AI policies.Manufacturing generates the most AI traffic with 20.9%of all AI/ML transactions in the Zscaler cloud,followed by Finance and Insurance(19.9%)and Services(16.8%).ChatGPT usage continues to soar,with 634.1%growth,even though it is also the most-blocked AI application by enterprises,based on Zscaler cloud insights.2024 Zscaler,Inc.All rights reserved.05ZSCALER THREATLABZ REPORT 2024Key GenAI and ML Usage TrendsAI transactions continue to accelerateFrom April 2023 to January 2024,enterprise AI and ML transactions grew by nearly 600%,rising to more than 3 billion monthly transactions across the Zero Trust Exchange in January.This underscores the fact that,despite a rising number of security incidents and data risks associated with enterprise AI adoption,its transformative potential is too great to ignore.Note that while AI transactions saw a brief lull over the December holidays,transactions continued at an even greater pace at the start of 2024.Even as AI applications proliferate,however,the majority of AI transactions are being driven by a relatively small set of market-leading AI tools.Overall,ChatGPT accounts for more than half of all AI and ML transactions,while the OpenAI application itself comes in third place,with 7.82%of all transactions.Meanwhile,Drift,the popular AI-powered chatbot,generated nearly one-fifth of enterprise AI traffic(the LivePerson and BoldChat Enterprise chatbots also breached the top apps in spots 5 and 6).Meanwhile,Writer remains a favored generative AI tool in the creation of written enterprise content,such as marketing materials.Finally,Otter,an AI transcription tool often used in video calls,drives a significant portion of AI traffic.4000M3000M2000M1000M0MMayJulSepMonthTransactionsNovJanChatGPTDriftOpenAIWriterLivePersonBoldChat EnterprisesOtter AI52.23%18.51%7.82%3.86%2.78%2.06%1.29%AI and ML Transaction TrendsTop AI ApplicationsThe enterprise AI revolution is far from its peak.Enterprise AI transactions have surged by nearly 600%and show no signs of slowing.Still,blocked transactions to AI apps have also risen by 577%.FIGURE 1 AI transactions from April 2023 to January 2024FIGURE 2 Top AI applications by transaction volume2024 Zscaler,Inc.All rights reserved.06ZSCALER THREATLABZ REPORT 2024Meanwhile,the volumes of data that enterprises send and receive from AI tools adds nuance to these trends.Hugging Face,the open-source AI developer platform often described as“the GitHub of AI,”accounts for nearly 60%of enterprise data transferred by AI tools.Since Hugging Face allows users to host and train AI models,it makes sense that it captures significant data volumes from enterprise users.While ChatGPT and OpenAI make expected appearances on this list,two notable additions are Veedan AI video editor often used to add subtitles,imagery,and other text to videosand Fotor,a tool used to generate AI images,among other uses.Since videos and images entail large file sizes compared to other kinds of requests,its not surprising to see these two applications represented.Enterprises are blocking more AI transactions than everEven as enterprise AI adoption continues to surge,organizations are increasingly blocking AI and ML transactions because of data and security concerns.Today,enterprises block 18.5%of all AI transactions,a 577%increase from April to January,for a total of more than 2.6 billion blocked transactions.Some of the most popular AI tools are also the most blocked.Indeed,ChatGPT holds the distinction of being both the most-used and most-blocked AI application.This indicates that despiteor even because ofthe popularity of these tools,enterprises are working actively to secure their use against data loss and privacy concerns.Another notable trend is that ,which has an AI-enabled Copilot functionality,is blocked from April to January.In fact, accounts for 25.02%of all blocked AI and ML domain transactions.Fotor0.8%VEED4.4%OpenAI4.7%ChatGPT27.9%Hugging Face57.1%Data transferred by AI/ML Traffic Sep 2023-Jan 2024800M600M400M200M0MMaySepMonthTransactionsNovJanJulBlocked AI transaction trends Apr 2023-Jan 2024KEY GENAI AND ML USAGE TRENDSFIGURE 3 Top AI/ML apps by the percentage of total data transferredFIGURE 4 Number of AI/ML transactions blocked over time2024 Zscaler,Inc.All rights reserved.07ZSCALER THREATLABZ REPORT 2024Industry AI breakdownEnterprise industry verticals show notable differences in their overall adoption of AI tools as well as the proportion of AI transactions they block.Manufacturing is the clear leader,driving more than 20%of AI and ML transactions across the Zero Trust Exchange.Still,the finance and insurance,technology,and services sectors follow closely behind.Together,these four industries have pulled ahead of others as the most aggressive AI adopters.TOP MOST-BLOCKED AI TOOLS01 ChatGPT02 OpenAI03 F04 Forethought05 Hugging Face06 ChatBot07 Aivo08 Neeva09 infeedo.ai10 JasperTOP BLOCKED AI DOMAINS01 B02 Divo.ai03 D04 Q05 Compose.ai06 O07 Qortex.ai08 Sider.ai09 T10 securiti.aiEducation1.7%Energy,Oil&Gas1.7%Government3.3%Retail&Wholesale4.9%Healthcare5.5%Others7.6%Technology15.6%Services16.8%Finance&Insurance19.9%Manufacturing20.9%800MManufacturingFinance&InsuranceTechnologyServicesRetail&WholesaleHealthcareGovernmentEducation600M400M200M0MMaySepNovJanJulShare of AI Transactions by Industry VerticalAI Transaction Trends by VerticalKEY GENAI AND ML USAGE TRENDSFIGURE 6 Industries driving the largest proportions of AI transactionsFIGURE 5 Top blocked AI applications and domains by volume of transactionsFIGURE 7 AI/ML transaction trends among the highest-volume industries,April 2023January 2024Vertical%of AI transactions blockedFinance&Insurance37.16Manufacturing15.65Services13.17Technology19.36Healthcare17.23Retail&Wholesale10.52Others8.93Energy,Oil&Gas14.24Government6.75Transportation7.90Education2.98Communication4.29Construction4.12Basic Materials,Chemicals&Mining2.92Entertainment1.33Food,Beverage&Tobacco3.66Hotels,Restaurants&Leisure3.16Religious Organizations6.06Agriculture&Forestry0.18Average across all verticals18.53FIGURE 8 Top industry verticals by percentage of AI transactions blocked2024 Zscaler,Inc.All rights reserved.08ZSCALER THREATLABZ REPORT 2024Securing AI/ML transactionsPaired with the sharp rise in AI transactions,industry sectors are blocking more AI transactions.Here,certain industries diverge from their overall adoption trends,reflecting differing priorities and levels of maturity in terms of securing AI tools.The finance and insurance sector,for instance,blocks the largest proportion of AI transactions:37.2%vs.the global average of 18.5%.This is likely due in large part to the industrys strict regulatory and compliance environment,combined with the highly sensitive financial and personal user data these organizations process.Meanwhile,manufacturing blocks 15.7%of AI transactions,despite its outsized role in driving overall AI transactions.The technology sector,one of the earliest and most eager adopters of AI,has taken something of a middle path,blocking an above-average 19.4%of AI transactions as it works to scale AI adoption.Surprisingly,the healthcare industry blocks a below-average 17.2%of AI transactions,despite these organizations processing a vast wealth of health data and personally identifiable information(PII).This trend likely reflects a lagging effort among healthcare organizations to protect sensitive data involved in AI tools,as security teams play catch-up to AI innovation.Overall AI transactions in healthcare remain comparatively low.KEY GENAI AND ML USAGE TRENDSPercent of Blocked AI Transactions by VerticalHealthcare and AIRanking as the sixth biggest AI/ML user,the healthcare industry blocks 17.23%of all AI/ML transactions.THE TOP AI APPS IN HEALTHCARE ARE:Vital signs of progress in AI healthcareWhile the healthcare industry is typically cautious when putting innovations like AI into practice,as seen by its current 5%contribution to AI/ML traffic in the Zscaler cloud,its only a matter of time before AI has a greater impact on healthcare operations,patient care,and medical research and innovation.1Indeed,AI promises to help not only save time,but also save lives.Already,AI-powered technologies are enhancing diagnostics and patient care.By analyzing medical images with remarkable accuracy,AI helps radiologists detect abnormalities more quickly and facilitates faster treatment decisions.2The potential benefits are vast.AI algorithms can use patient data to personalize treatment plans and accelerate drug discovery by efficiently analyzing biological data.Administrative tasks can be automated with generative AI as well,alleviating burdens on short-staffed healthcare teams.These advancements underscore AIs capacity to transform health provision and healthcare delivery.01 ChatGPT02 Drift03 OpenAI04 Writer05 Intercom06 Zineone07 Securiti08 Pypestream09 Hybrid10 VEED2024 Zscaler,Inc.All rights reserved.09ZSCALER THREATLABZ REPORT 2024Key Healthcare Risks:Healthcare organizations should acknowledge the potential risks and challenges associated with AI,including concerns about data privacy and security,especially for personal identifiable information(PII),as well as ensuring that AI algorithms and their outputs are highly reliable and unbiased when aiding in the administration of patient care.1.Statista,Future Use Cases for AI in Healthcare,September 2023.2.The Hill,AI already plays a vital role in medical imaging and is effectively regulated,February 23,2024.KEY GENAI AND ML USAGE TRENDS2024 Zscaler,Inc.All rights reserved.010ZSCALER THREATLABZ REPORT 2024Finance&AIIn second place for total AI/ML usage,the finance industry blocks 37.16%of all AI/ML traffic.THE TOP AI APPS IN FINANCE ARE:Financial institutions bank on AIFinancial services