ChatGPT test shows how AI can be fooled

There’s more evidence that ChatGPT won’t put IT security teams out of work — yet.

Researchers at Endor Labs tested ChatGPT 3.5 against 1,870 artifacts from the PyPi and NPN open-source repositories of code. It identified 34 as having malware. However, only 13 really had bad code. Five others did have obfuscated code but did not expose any malicious behavior, while one artifact was a proof-of-concept that downloads and opens an image via  an NPM install hook. As a result, the researchers considered ChatGPT-3.5 right 19 out of 34 choices.

However, 15 of the results were false positives.

The researchers also found the version tested can be tricked into changing an assessment from malicious to benign by using innocent function names, including comments in a query that indicate benign functionality or through the inclusion of string literals.

Large-language model-assisted malware reviews “can complement, but not yet substitute human reviews,” Endor Labs researcher Henrik Plate concluded in a blog.

However, the most recent version is ChatGPT-4, which Plate acknowledged gave different results.

And, he admitted, pre-processing of code snippets, additional effort on prompt engineering, and future models are expected to improve his firm’s test results.

Researchers say large language models (LLMs) such as GPT-3.5 or GPT-4 can help IT staff assess possible malware. Microsoft is already doing that with its Security CoPilot application.

Still, the researchers’ conclusion is: ChatGPT-3.5 isn’t ready to replace humans.

“One inherent problem seems to be the reliance on identifiers and comments to ‘understand’ code behavior,” Plate writes. “They are a valuable source of information for code developed by benign developers, but they can also be easily misused by adversaries to evade the detection of malicious behavior.

“But even though LLM-based assessment should not be used instead of manual reviews, they can certainly be used as one additional signal and input for manual reviews. In particular, they can be useful to automatically review larger numbers of malware signals produced by noisy detectors (which otherwise risk being ignored entirely in case of limited review capabilities).”

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

Howard Solomon
Howard Solomon
Currently a freelance writer, I'm the former editor of and Computing Canada. An IT journalist since 1997, I've written for several of ITWC's sister publications including and Computer Dealer News. Before that I was a staff reporter at the Calgary Herald and the Brampton (Ont.) Daily Times. I can be reached at hsolomon [@]

Featured Articles

Cybersecurity in 2024: Priorities and challenges for Canadian organizations 

By Derek Manky As predictions for 2024 point to the continued expansion...

Survey shows generative AI is a top priority for Canadian corporate leaders.

Leaders are devoting significant budget to generative AI for 2024 Canadian corporate...

Related Tech News

Tech Jobs

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

Tech Companies Hiring Right Now