AI's Promise and Limits: GPT-4 Skilled in Code Analysis but Inadequate for Security Audits, Salus Study Reveals
Summary:
A study by researchers at Salus Security reveals that artificial intelligence system GPT-4 shows promise in creating and analyzing code but falls short in security auditing. Using 35 smart contracts from the SolidiFI-benchmark vulnerability library, it was found to achieve 80% precision in identifying true positives, though its recall rate was at a low 11%. The researchers recommend the continued use of traditional auditing methods and tools along with AI systems for improved accuracy and efficiency in smart contract audits.
Researchers at Salus Security, a global blockchain security firm, have recently disclosed findings from a study focusing on the capabilities of GPT-4, an artificial intelligence (AI) system. They found that while it is adept at creating and analyzing coding language, it is not yet suitable for use as a security examiner.
According to the published paper, GPT-4 has potential benefits for aiding smart contract audits, primarily in code analysis and providing vulnerability tips. Nevertheless, due to its restricted ability in identifying vulnerabilities, it cannot fully supplant expert auditing applications and seasoned auditors presently.
To gauge the AI's ability to identify potential security flaws, the team at Salus utilized a total of 35 smart contracts from the SolidiFI-benchmark vulnerability library which displayed a total of 732 vulnerabilities, and examined seven prevalent types of vulnerabilities.
Their research demonstrated that the ChatGPT system is proficient at identifying true positives - genuine vulnerabilities which warrant further investigation in a real-world context. It exhibited over 80% precision in tests. But, it does suffer from a significant issue in generating false negatives. This aspect is quantified by a metric known as "recall rate". In the experiments conducted by the team, the recall rate for GPT-4 was as low as 11% (the higher, the better).
Such results led the researchers to conclude that GPT-4 has limited vulnerability detection abilities, achieving a peak accuracy of merely 33%. Thus, the use of specialized audit tools and conventional human expertise remains recommended for smart contract audits until AI systems like GPT-4 have improved.
In conclusion, while GPT-4 can provide support during smart contract audits, especially with code analysis and vulnerability cues, its use still needs to be backed by other audit practices and tools to enhance the thoroughness and validity of the audit process.
Published At
2/20/2024 7:41:30 PM
Disclaimer: Algoine does not endorse any content or product on this page. Readers should conduct their own research before taking any actions related to the asset, company, or any information in this article and assume full responsibility for their decisions. This article should not be considered as investment advice. Our news is prepared with AI support.
Do you suspect this content may be misleading, incomplete, or inappropriate in any way, requiring modification or removal?
We appreciate your report.