Top 5 Signs Your Current Penetration Testing Provider Is Underdelivering
A security manager at a mid-sized fintech company once described her annual penetration test as "the world"s most expensive PDF." The report arrived every October. The findings looked familiar. Leadership ticked the box. The file went into a folder. And the following October, the same process repeated.
She was not wrong to feel something was off. She just did not have the language to name it.
What she was experiencing is one of the most common and least-discussed problems in corporate security: a penetration testing programme that is technically functioning but not actually protecting the business at the level it should. The provider is not missing deadlines. The reports are not full of errors. Everything looks fine on paper. And that is precisely the problem.
Capture The Bug has assessed environments where businesses had years of consistent penetration testing from credentialed providers, and still had significant undetected exposure. The reports existed. The compliance box was ticked. The attack paths were still open.
These are the five signs that a penetration testing provider is underdelivering, and what each one means for the business relying on them.
Sign One: The Report Reads Like It Was Written for Someone Else
The fastest way to assess a penetration testing engagement is to read a single finding carefully. Not the summary page. Not the severity chart. One finding, read closely.
A finding from a rigorous engagement tells a specific story. It describes the exact vulnerability in the exact environment being tested. It includes evidence that confirms it was discovered and demonstrated. It explains what an attacker could actually do with it. And it provides remediation guidance specific enough for an engineer to act on without needing to go back and ask clarifying questions.
A finding from a template-driven engagement describes a vulnerability category. The language could apply to any web application, any API, any server environment. The evidence section is sparse. The remediation guidance is a one-liner that references patching or a version update. There is no business context and no explanation of what the impact would actually look like in that specific organisation.
When every finding in a report shares the same structure, the same tone, and the same generic language regardless of the vulnerability type, the report is not the product of genuine expert testing. It is a formatted document built around surface-level assessment with familiar language filled in around it. The distinction matters because a finding that cannot be acted on is not a finding. It is a line item.
Sign Two: Nothing in the Report Surprises Anyone
Capture The Bug hears a version of this regularly when speaking with security leaders who are transitioning from another provider. When asked what the most significant finding from their last engagement was, the answer is almost always something the team already knew, or something that appeared in the previous year's report as well.
A penetration test that only confirms what the internal team already knows is not adding security value. It is producing confirmation of existing documentation.
This happens because genuine penetration testing requires an external perspective, different techniques, and adversarial thinking applied to the specific environment. The people who built an application know how it is supposed to work. A skilled external tester discovers how it actually behaves when someone is deliberately trying to make it fail.
When the findings are predictable year after year, when no one reading the report encounters something they did not already have on their internal list, the engagement is not discovering new attack surface. It is running a familiar process against a familiar target. Rigorous testing should regularly produce findings that prompt real conversations, that cause the team to reconsider assumptions, and that surface attack paths that internal review did not catch. That is the standard against which every engagement should be measured.
Sign Three: Remediation Support Disappears After Report Delivery
Delivering the report is not the end of the engagement. For providers who treat it that way, a significant portion of what the client paid for is never realised.
A finding that is documented but not correctly fixed is still a vulnerability. The documentation exists. The exposure does not go away.
The patterns that appear most often in underdelivering engagements are predictable. The provider delivers the report and then becomes difficult to reach when engineers have clarification questions. The remediation guidance is too generic to implement without additional research. Retesting is either not offered, charged separately in a way that discourages businesses from requesting it, or treated as optional rather than standard.
When engineering teams receive findings they cannot act on clearly, and the provider is not available to support them, remediation stalls. Findings get marked as resolved on an internal tracker because something adjacent to the recommendation was implemented, not because the attack path was actually closed.
Capture The Bug builds remediation support and retest verification into every engagement at https://capturethebug.xyz/Services/penetration-testing because the value of identifying a vulnerability is only realised when the fix is confirmed by the same team that found it. A retest conducted independently of the original finding is the only reliable way to confirm that the remediation addressed the full attack path rather than just the surface presentation of it. If a provider has never offered a retest, or has positioned retesting as an optional upgrade, the engagement model is not designed around security outcomes.
Sign Four: The Scope Has Not Changed in Years
A penetration testing scope that has remained identical across three or four consecutive years is not evidence of a disciplined programme. It is evidence of a programme that stopped keeping pace with the business it is supposed to protect.
Most technology companies change significantly over twelve months. New features are shipped. New third-party integrations are onboarded. Infrastructure moves to new cloud environments. APIs that did not exist two years ago now process significant volumes of sensitive data.
When the scope does not change, the test assesses an accurate picture of what the environment looked like when the scope was first defined, not what it looks like today. The areas of highest current risk may not be covered at all.
A provider that does not actively review and update scope before each engagement is not engaging with the current risk profile of the business. They are delivering a familiar service against a familiar target because that is operationally convenient for both sides. The conversation that should happen before every engagement is not "same as last year?" It should be "what has changed, and where does that put the highest risk today?" Capture The Bug treats scope review as a required part of every engagement kickoff, not a formality.
Sign Five: The Provider Cannot Explain How Their CREST Certification Applies
CREST certification is not a logo on a website. It is an internationally recognised standard that verifies the methodology, the people, and the quality controls behind an engagement meet a defined benchmark.
In the Australian and New Zealand market, APRA-regulated entities, CDR participants, and enterprise clients increasingly require penetration testing from CREST-certified providers. Similar expectations exist across regulated sectors in the United States.
A provider with CREST certification should be able to answer directly: which testers on this engagement hold individual CREST qualifications, how does the certification methodology apply to this specific type of test, and what is the quality assurance process between the tester and the final report.
If those questions produce vague answers, or if the certification appears on the company credentials page but the individuals actually running the test hold no individual qualifications, the certification is not being applied to the engagement in the way it is implied.
For businesses that need to evidence compliant security testing to a regulator, an auditor, or an enterprise client, this distinction matters considerably. A report produced by a CREST-registered company where the engagement was not conducted to CREST methodology does not carry the weight the certification implies. Every assessment through https://capturethebug.xyz/Services/penetration-testing is delivered under the methodology and quality standards that CREST certification requires, with testers who hold the individual qualifications to support that claim.
What to Do With This Information
The five signs above rarely arrive all at once. They accumulate quietly over months or years while the business continues to believe its security programme is functioning as it should.
Reports that no one reads twice. Findings that confirm what was already known. Remediation guidance that engineering teams cannot act on. A scope that reflects the business from three years ago. Certification claims that dissolve under a direct question. Any one of these is worth a conversation with the current provider. More than one is worth a harder look at whether the investment being made in security testing is delivering the protection it should.
The businesses that Capture The Bug works with come from different starting points. Some have never had a formal penetration test. Others have had years of consistent testing that never went deep enough to matter. In both cases, the work starts in the same place: a scope review, a qualified team, and an engagement process designed around finding real vulnerabilities rather than producing familiar documentation.
A security programme that is genuinely working finds things that matter, drives remediation that gets verified, and produces evidence that holds up when regulators, boards, and enterprise clients ask for it.
If the current programme is not doing those three things, the question is not whether to make a change. It is how long the existing exposure has been sitting there undetected.
Plan Your Annual Pentesting Strategy the Right Way
Learn how modern SaaS companies structure pentesting across the year to reduce risk, stay compliant, and avoid last-minute panic before audits.
Frequently Asked Questions
How can I tell if my penetration testing report is high quality or just a template?
Read one finding closely. A high-quality finding is specific to the environment being tested. It includes verified evidence, explains what an attacker could actually do with the vulnerability, and provides remediation guidance that an engineer can act on. A template finding uses generic language that could apply to any system and gives a one-line remediation instruction with no business context.
Should penetration testing findings ever surprise the internal security team?
Yes, regularly. An external tester brings objective thinking and a fresh perspective that internal familiarity with a system obscures. When every finding in a report is already on the internal list or identical to the previous year's report, the engagement is confirming existing documentation rather than discovering new attack surface. That is not what penetration testing should deliver.
Should remediation support be included in a penetration testing engagement?
Yes. Remediation support and retest verification should be standard, not optional. A finding that is documented but not correctly fixed is still a live vulnerability. Retesting by the same team that identified the vulnerability is the only reliable way to confirm the fix closed the full attack path, not just the surface symptom.
How often should penetration testing scope be reviewed and updated?
Before every engagement. Most businesses change significantly over twelve months through new features, integrations, infrastructure migrations, and new data flows. A scope that does not reflect the current environment produces a test that assesses yesterday's risk, not today's. Scope review should be a required part of every engagement kickoff.
What questions should I ask a penetration testing provider about their CREST certification?
Ask which testers on the engagement hold individual CREST qualifications, how the CREST methodology applies to the specific type of test being conducted, and what the quality assurance process is before the report reaches the client. A CREST-certified company should answer all three clearly. If the certification is on the website but the individuals running the test hold no individual qualifications, the certification is not being applied at the engagement level.
How many of these signs need to be present before switching penetration testing providers?
Any single sign warrants a direct conversation with the current provider. More than one suggests the engagement model is not designed around security outcomes. A combination of template reports, predictable findings, no remediation support, a static scope, and unclear certification standards indicates the programme is producing compliance documentation rather than genuine security protection.
What does a CREST-certified penetration testing engagement actually require?
CREST certification at the engagement level means the methodology, documentation standards, evidence requirements, and reporting quality all meet the benchmarks CREST specifies. Individual testers should hold relevant CREST qualifications for the type of testing being conducted. A quality review process should sit between the tester and the final report. This applies to the specific engagement, not just to the company's registration status.
