Why Traditional AI Search Engines Fail Security Engineers

There is no one right answer in security.

It stems from the fact that there is no such thing as 100% security, "is the juice worth the squeeze" is a question we always ask in security, "is the effort of fixing the security issue worth the risk that the fix potentially reduces".

"There is no right answer" is more true in compliance. Many compliance standards are intent based. As long as you can make a case to the auditor that you are staying true to the intent of what the actual line says in the standard, you will generally be fine.

Traditional AI search engines crawl websites, put the content in RAG and tries to answer the question using content which closely matches the user query. Most of the links are not from official standard documentation, and are from vendor blogs. Worse yet, many of the links are out dated, "page not found".

Search engines don't care about source attribution, attributing the answers to the right sources, and definitely don't understand nor care about the compliance intent when answering the user question.

Lets try this simple question of 'what version of TLS do i need to have for PCI in Perplexity. Perplexity gives the following answer : "the most current guidance recommends 1.2'.

Now if you are a compliance engineer, reading through the answer, you are trying balance the organizations compliance vs making the organization go through all the dev and infrastructure work to make it TLS 1.2 happen. You want to know which official document (PCI official compliance standard document) says that and what is the intent of the compliance (PCI official guidance documents) and how it gets tested (audit test document if any).

By the way, if you read through myriad of official PCI documents , no where it says "you must use 1.2 and you must not use 1.1"

Here is the workflow that we have heard from compliance engineers after they search -

1. Click on each of the links that the search engine provides and read through them, take out the vendor bias in your mind
2. Find all the official documents for the standard, find the words TLS or SSL or crypto, read through them
3. Take screenshots, notes , form an informed opinion so you can justify your opinion to the broader team

Today, we updated Transilience AI Cyber Consultant app, we went about solving exactly that.

The app now gets all the official documents, so you dont have to. The app fetches the corresponding pages right inline so you have to search inside the document. The app highlights the right text so you dont have to look for the text.

Lets try it. Putting the same question, I get the same answer but with official source attribution .

The app picks about half a dozen official PCI documents. Interspersed with the documents, the app gives the relevant text why we picked the document

The app then highlights the relevant text inside each of the pages

Going through the pages, with in a few minutes, the user can get to the intent and source attbution.

Inside PCI quick reference guide, PCI states to use strong cryptography

PCI report on compliance document states that early versions of TLS have vulnerabilities and must have risk mitigation plans for vulnerabilities associated with early version of vulnerabilities

Equipped with official sources and wording, compliance engineer can confidently conclude the what is the intent behind each of the official PCI documents to make the right decisions for the organization.

Happy researching.

Transilience AI backend engineering team - Smritika Sadhukhan ✨ Aman Agarwal Alessio Mauro