Harmful Artificial Intelligence Models: Risks and Protection Ways

Short response: Harmful artificial intelligence models are restructured language models for security filters removed or abuse, and vehicles working with these models. They are most often used to accelerate e-mail, fraud text, dezenformation and harmful software production. The basis of protection is to verify the messages that want to urgent money or password from a second channel, use multi-factor authentication and establish artificial intelligence systems according to the security guides of institutions such as OWASP, NCSC.

Last update: 14.10.2026

With the developing technology, the potential for abuse is increasing, as well as the conveniences of artificial intelligence to our lives. malicious uses can accelerate many cyber threats from intrusion attacks to spying, malware production without dezenformation. In this article, examples of security firms and researchers report are telling the European Union’s Artificial Intelligence Act and how to protect itself with official sources. We do not provide any methods or access information for misuse.

What Is Harmful Artificial Intelligence Models?

Harmful artificial intelligence models are customized to be used in illegal activities such as cyber criminals fraud and manipulation. Mostly open source these models are trained and replaced system prompts k “jailbreak” created versions. In this process, ethical and security filters are disabled and ensure that the model responds all kinds of request. Thus, artificial intelligence is trained with content such as fraud scenarios, malicious software codes, hate phrases by personalizing in malicious branches.

Reported examples at a glance

Language model trained with toxic forum posts

2022, Yannic Claycher

Hugging Face closed access

FC II

The worm scenario emitted between AI assistants

March 2024, academic research

Shows in controlled environment, real attack w l

Home

examples of ransomware using artificial intelligence model

26.08.2025, ESET

Proof of concept was not seen in real attack

DeepSeek-R1

Meşru outdoor model was poor in safety tests

31.01.2025, Cisco and Pennsylvania University

The risk of misuse was shown in safety test

FraudGP Home

FraudGPT, Netenrich threatened research team at 25.07.2023 defined is a artificial intelligence chat tool that is marketed for only attack purposes. The report states that the vehicle is introduced in cybercrime markets and messaging channels. The following talents were highlighted in the promotion according to the researchers:

• Ability to write ID hunting emails, login messages

• Help to produce malicious code or malware

• Providing recommendations on credit card fraud and fake website preparation

• Safety filters and ethical limitations

Tags Home

WormGPT was introduced as a non-security chat service in cyber criminal forums in the summer of 2023. Security firm SlashNext wrote a convincing e-mail that can be used in the test of July 2023, to convince employees to pay a fake bill, in the fraudulent e-mail scam (BEC). Krebs on Security 08.08.2023 According to the developer said that after the demands of abuse, the service becomes more controlled. Featured risks:

• Ability to produce high-level social engineering/chemistry hunting texts

• Generate content compatible with spam/filter cutting techniques

• Ability to organize large measured attacks even users with low technical knowledge

• Even if a vehicle changes or closes, the same method can be repeated with other means

GPT-4Chan

A YouTube content manufacturer named Yannic Kilcher, which is among the artificial intelligence models used to the evil, has been trained with the message that contains millions of toxic and hatred phrases. The model was trained by about 3.5 years in 4chan/pol/board. Although it is offered as an academic experiment, the outputs carry serious ethical and social risks Hugging Face closed access to model. Basic properties are as follows:

• The model showed that weights can be freely shared, copied and re-trained by such models.

• It has the potential to produce any harmful, toxic or illegal content.

• The content contains racism, sexism, antisemitism and conspiracy theories.

• It can be used by cyber criminals to produce text and social engineering scenarios.

FC II

Morris II is a security research, not an attack tool or model. The name spreads to the internet in 1988 CiraganReceives from . Stav Cohen, Ron Bitton and I Nassi “Here Comes The AI Worm” work (the first version of March 2024) showed itself in controlled environment where a malicious prompt can be spread without user clicks between productive AI-supported email assistants. The highlights of the study are:

• Attack was shown as chaining spread scenario among AI applications that read external content (using RAG).

• It was reported that the affected assistant is forced to leak confidential data in every step.

• The researchers suggested a protection called “Virtual Donkey” in the same study; they reported 0,015 false positive rates in tests.

Home

PromptLock ESET researchers announced at 26.08.2025The first known ransom software example using artificial intelligence model. She was producing her scripts with an open-weight language model that works locally. ESET noted that for example, it was not seen in real attacks and is a proof of concept; a academic team at 03.09.2025 reported ESET that research prototypes called “Ransomware 3.0”. Featured points:

• It was not seen in a real attack; a prototype for research purposes.

• It is an early warning that artificial intelligence can be used to automate threats known as ransomware.

• Classic measures such as regular backup, up-to-date software and endpoint protection protect the validity of such threats.

DeepSeek-R1

DeepSeek-R1, based in China DeepSeek Despite being an open sourced productive language model developed by the company, security filters are disabled or misused harmful potential can carry. Cisco and Pennsylvania University researcher published in 31.01.2025 The model did not block any of the 50 malicious request selected from the HarmBench data set (100% attack success rate on the earth); this rate in the same test was 26 percent for OpenAI o1-preview. Risks designed for research, software development and text production may lead when misuse of the model are used:

• It leads to harmful code production by creating software, viruses or spy software that exploit security open.

• Impressive fake emails and social engineering scenarios can be produced.

• Fake news, manipulative content and propaganda texts can be created.

Protection Methods Against Harmful Artificial Intelligence Models

• Creating Safety Layers: Prevent direct and indirect injections using prompt filtering systems and prevent the manipulation of the model with firewalls that analyze user entries.

• Model Training and Inspection Home: With Adversarial testing, testing the durability of the model against abuse scenarios and making toxic, misleading and harmful content in educational data.

• User and Corporate Awareness: Media literacy can be improved in recognition of content produced by artificial intelligence and authorization and access limits can be determined in the use of internal AI.

• Technical Tools and Solutions: Technical solutions for cybercrime are available. These are:

AI Red Team g: Identifying the open by simulating artificial intelligence abuse scenarios.

Model watermarkin g: Some manufacturers add an invisible sign to the content they produce; for example, Google, SynthID Detector watermarks on the content produced in their own vehicles. Filigran can only be verified by the tools of the same manufacturer, does not give precise results for unwatermarked content. For Details AI content detection tools See our article.

Zero-trust architecture: Continuous verification and inspection should be applied in access to artificial intelligence systems.

Prompt injection, The first 10 risk list for OWASP’s 2025 LLM applications ranked first (LLM01). In this attack, the model entered or a web page, e-mail or document that the model reads, changes the behavior of the model in the direction that the developer does not want. In the list there are risks such as sensitive information disclosure (LLM02), data and model poisoning (LLM04) and excessive authorization (LLM06). If you are giving permissions such as sending email to AI assistant, payment or file deletion, you must take into account these risks.

Legal framework: EU AI Law and Turkey

European Union Artificial Intelligence Act (Thirk (AB) 2024/1689) 01.08.2024 entered into force and gradually applied. The law also covers companies used in the EU, offering artificial intelligence system to the EU market, so the initiatives that serve from Turkey to EU are also interested.

Requirements for governance rules and general purpose artificial intelligence (GPAI) models

August 2026

Transparency rules: deepfake and artificial intelligence production to be labeled in the visible form of content, the report that conversation bots are talking with the machine

December 2026

9. ban added with AI Omnibus

02.12.2027

High risk systems in sensitive areas (Ex III)

02.08.2028

High risk systems integrated into products (Ex I)

The dates for high-risk systems were delayed by the “AI Omnibus” regulation, which was proposed in 19.11.2025 and entered into force in 27.07.2026. The same arrangement according to the European Commission also brings a ban on applications that produce non-reactive image.

The main frame of cybersecurity in Turkey is released in 19.03.2025 in the Official Gazette Cyber Security Law No. 7545 u. Based on Cyber Security Presidency National Cyber Events Intervention Centre (USOM); cyber ihbars receive, publish malicious link lists and security notifications.

Checklist to protect yourself and organization

For individual users

• Verify messages that want emergency money, code or password, even if the sender looks like a familiar one, from a second channel (such as call back to phone).

• Open multi-factor authentication in email, bank and social media accounts.

• Identify a verification question that you know alone against voice or image plug.

• Report suspicious links and cyber events to USOM; immediately call the bank if you have been scammer and refer to the prosecutor or colleague.

For companies and developers

• NCSC and CISA’s Safe Artificial Intelligence System Development GuideApply four areas in (27.11.2023): safe design, safe development, safe distribution, safe operation and maintenance.

Frequently Asked Questions

What is the harmful artificial intelligence model?

Safety filters are restructured artificial intelligence models and tools for removed or abuse. They can be used to speed up the production of e-mail, fraud text, dezenformation and malicious code.

Do FraudGPT and WormGPT still exist?

Today’s situations cannot be verified by an official source; two were reported by security firms in 2023. Not important names, the method they use: convincing and fraud messages. Protection routes remain the same.

When does EU AI Act apply?

The law entered into force in 01.08.2024 and is applied progressively: prohibited from 02.02.2025, rules for general purpose models are valid from 02.08.2025, transparency rules from August 2026. High risk system rules were delayed to 02.12.2027 and 02.08.2028.

Where can I report scams made with artificial intelligence?

You can report cyber events and malicious links to USOM based on Cyber Security Presidency. Call the bank immediately if there is no loss, then complain to the prosecutor or arm.

How do I understand that a text or visual is produced with artificial intelligence?

There is no exact method; detection tools are estimated and can be burned. Filigran verification gives more reliable results when used together to confirm the source control and content from a second channel. Limits of vehicles AI content detection software We spoke in our article.