How security metrics are verified in testing phase?

Question

I am required to analyze and make an architecture of an application. While analyzing the requirements I find, in my system user personally identifiable information (PII) confidentiality is a very sensitive quality requirement and it must be taken to the NFR (non functional requirement) section. NFR requires that all the responses should have measurable equation which is used by tester in verification phase.

I am new in this area and facing hurdle for preparing data security related measure and their allowance limit. I prepared the NFR table as below and I am afraid that might not be proper way because, in verification phase it will be tough for tester to verify this requirement.

So, the question is, how usually data confidentiality and security attribute measures and allowance limits are set by architects for such software verification phase by testers?

Answer 1

As you've already surmised, a metric like "Attack Success Rate <= 0.001%" isn't a particularly useful test metric for an acceptance text.

Have a look at this page. It says:

Non-functional requirements

Security

Provider systems SHALL resist unauthorised, accidental or unintended usage and provide access only to legitimate users. Please refer to the Security guidance page for technical details.

Now, of course, that's not a testable requirement; it is a wish. However, if you go to the Security guidance page that is linked there, you will find (among other things) this:

Secure connection negotiation

Provider systems:

• SHALL only accept connections from the Spine Secure Proxy (SSP)

• SHALL authenticate the SSP prior to responding to any requests using its client certificate

• SHALL only permit approved supported ciphers to be utilised

• SHALL only accept encrypted connections and drop connection attempts presented over insecure protocols

• SHALL only accept requests for its allocated address space identifier (ASID), as specified by the Ssp-To header on its matching endpoint URL

• SHALL check that the Ssp-InteractionID value is consistent with the endpoint being requested

• SHALL check for the presence of all SSP headers

• SHALL check that an authorization bearer token is present and correctly formed

• MAY authorise access to API endpoints through examining acceptable values in the JSON Web Tokens (JWT) requested_scope claim

• SHALL risk-manage the security of the endpoints of the Transport Layer Security (TLS) communications, so as to prevent inappropriate risks (for example, audit logging of the GET parameters into an unprotected audit log)

Ah, now we're getting somewhere. For the most part, these are testable requirements. Not only are they specific and testable, they also provide a detailed outline of your security strategy.

Answer 2

NFR requires that all the responses should have measurable equation which is used by tester in verification phase.

I thinking this is probably the root cause of your issue. Some problems are just hard, and can't easily be reduced to a simple metric.

If your business truly cares about protecting its users' PII, the best thing to do is almost certainly to get it tested by a professional penetration testing service, and then set the metric to something like "all security vulnerabilities above trivial threat level have been resolved".