Comment pouvez-vous assurer le codage sécurisé avec Test Driven Development?

Question

Je viens à la vitesse sur la dernière tendance est le développement piloté par les tests (TDD). La plupart du développement que je fais est en C ou C ++. Il me semble qu'il ya un conflit évident entre les pratiques communes TDD et communes les pratiques de codage sécurisées. À son cœur, TDD vous dit que vous ne devriez pas écrire un nouveau code pour quelque chose que vous ne disposez pas d'un test de défaut pour. Pour moi, cela signifie que je ne devrais pas écrire du code sécurisé à moins d'avoir des tests unitaires pour voir si mon code est sécurisé.

Cela soulève deux questions:

1. Comment puis-je efficacement les tests unitaires écrire à tester pour le tampon Débordements, pile Dépassements, Heap Dépassements, erreurs Index Array, Bugs chaîne de format, ANSI vs Unicode vs taille de la chaîne MBCS mistmatches, une manipulation à cordes en toute sécurité (de Howard et LeBlanc "Writing Secure code")?

2. A quel point la pratique courante TDD doit ces tests inclus car une grande partie de la sécurité est. Non fonctionnel

Étonnamment, j'ai trouvé très peu de recherches discuter TDD et la sécurité. La plupart de ce que je viens à travers sont des documents TDD qui mentionnent à un très haut niveau que TDD va « rendre votre code plus sûr. »

Je cherche des réponses directes aux questions ci-dessus, toute recherche qui se rapporte à ce (je regardais déjà et ne trouve pas grand-chose), ou tout autre endroit qui vivent gourou TDD donc je peux aller frapper à leur porte ( virtuellement) et voir s'ils ont des bonnes réponses.

Merci!

EDIT:

Le thème de fuzzing est venu, que je pense est une bonne approche à ce problème (en général). Cela pose les questions suivantes: Est-ce que fuzzing se intégrer dans TDD? Lorsque, dans le processus TDD ne fuzzing en forme?

paramétrés tests unitaires (peut-être automatisé) a traversé mon esprit. Cela pourrait être un moyen d'obtenir des résultats fuzzing comme plus tôt dans le processus de test. Je ne sais pas exactement où cela se inscrit dans TDD soit.

EDIT 2:

Merci à vous tous pour vos réponses à ce jour. À ce stade, je suis très intéressé par la façon dont nous pouvons tirer parti des tests paramétrés pour servir de pseudo fuzzers pour nos fonctions. Mais, comment pouvons-nous déterminer les tests à écrire pour la sécurité test? Et comment pouvons-nous être sûrs que nous couvrons de manière adéquate l'espace d'attaque?

Il est un problème bien connu dans la sécurité des logiciels que si vous protégez contre 5 scénarios d'attaque, l'attaquant va simplement chercher et à utiliser, une 6e attaque. Il est un jeu du chat et de la souris très difficile. Est-ce que TDD nous donne un avantage contre cela?

Answer 1

Yes, TDD is a tool/technique that can help to ensure secure coding.

But as with all things in this industry: assume it's a silver bullet, and you'll shoot yourself in the foot.

Unknown Threats

As you indicated in Edit 2: "you protect against 5 attack scenarios, the attacker will just look for, and use, a 6th attack". TDD is not going to protect you from unknown threats. By its very nature, you have to know what you want to test in order to write the test in the first place.

So suppose threat number 6 is discovered (hopefuly not due to breach, but rather internally due to another tool/technique that attempts to find potential attack vectors).

TDD will help as follows:

• Tests can be written to verify the threat.
• A solution can be implemented to block the threat, and quickly be confirmed to be working.
• More importantly, provided all other tests still pass, you can quickly verify that:
  • All other security measures still behave correctly.
  • All other functionality still behaves correctly.
• Basically TDD assists in allowing a quick turnaround time from when a threat is discovered to when a solution becomes available.
• TDD also provides a high degree of confidence that the new version behaves correctly.

Testable Code

I have read that TDD is often misinterpreted as a Testing Methodology, when in fact it is more of a Design Methodology. TDD improves the design of your code, making it more testable.

Specialised Testing

An important feature of test cases is their ability to run without side-effects. Meaning you can run tests in any order, any number of times, and they should never fail. As a result, a number of other aspects of a system become easier to test purely as a result of the testability. For example: Performance, Memory Utilisation.

This testing is usually implemented by way of running special checks of an entire test suite - without directly impacting the suite itself.

A similar security testing module could overlay a test suite and look for known security concerns such as secure data left in memory, buffer overruns or any new attack vector that becomes known. Such an overlay would have a degree of confidence, because it has been checked for all known functionality of the system.

Improved Design

On of the key design improvements arising as a side-effect of TDD is explicit dependencies. Many systems suffer under the weight of implicit or derived dependencies. And these would make testing virtually impossible. As a result TDD designs tend to be more modular in the right places. From a security perspective this allows you to do things like:

• Test components that receive network data without having to actually send it over the network.
• One can easily mock-out objects to behave in unexpected / 'unrealistic' ways as might occur in attack scenarios.
• Test components in isolation.
• Or with any desired mix of production components.

Unit Testing

One thing that should be noted is that TDD favours highly localised (unit testing). As a result you could easily test that:

• SecureZeroMemory() would correctly erase a password from RAM.
• Or that GetSafeSQLParam() would correctly guard against SQL injection.

However, it becomes more difficult to verify that all developers have used the correct method in every place that it's required.
A test to verify a new SQL related feature would confirm that the feature works - it would work just as well with both the 'safe' and 'unsafe' versions of GetSQLParam.

It is for this reason you should not neglect other tools/techniques that can be used to "ensure secure coding".

• Coding Standards
• Code Reviews
• Testing

Answer 2

I'll take your second question first. Yes, TDD works can be used non-functional requirements. In fact, is often used as such. The most common benefit of an improved modular design, which is non-functional-- but seen by everyone who practices TDD. Other examples that I've used TDD to verify: cross-platform, cross-database, and performance.

For all your tests, you may need to restructure the code so that it is testable. This is one of the biggest effects of TDD-- it really changes how you structure your code. At first it seems like this is perturbing the design, but you soon come to realize that the testable design is better. Anyway...

String interpretation bugs (Unicode vs. ANSI) are particularly nice to test with TDD. It's usually straightforward to enumerate the bad and good inputs, and assert about their interpretation. You may find that you need to restructure your code a bit to "make it testable"; by this I mean extract methods that isolate the string-specific code.

For buffer overruns, making sure routines respond properly if given too much data is pretty straightforward to test as well. Just write a test and send them too much data. Assert that they did what you expected. But some buffer overflows and stack overflows are a bit trickier. You need to be able to cause these to happen, but you also need to figure out how to detect whether they happened. This may be as simple as allocating a buffers with extra bytes in them and verifying that those bytes don't change during tests... Or it may some other creative techniques.

I'm not sure there's a simple answer, though. Testing takes creativity, discipline, and commitment, but is usually worth it.

• isolate the behavior you need to test
• make sure you can detect the problem
• know what you want to happen for the error case
• write the test and see it fail

Hope this helps

Answer 3

TDD is the best way to build a secure system. All software developed by Microsoft is fuzzed and this arguably the number one reason for the dramatic reduction in vulnerabilities found. I highly recommended using the Peach Framework for this purpose. I have personally used Peach with great success in finding Buffer Overflows.

Peach pit files provide a way of describing the data used by your application. You can choose what interface you want test. Does your application read files? Does it have an open port? After you tell peach what the input looks like and how to communicate with your application, you can turn it loose and i knows all of the nasty input to make your application puke all over its self.

To make everything run, peach has a great testing harness, If your application crashes, peach will know because it has a debugger attached. When your application crashes, peach will restart it and keep testing. Peach can categorize all of the crashes and match up the core dumps with the input it used to crash the application.

Answer 4

Parameterized Tests

While we aren't doing buffer overrun test at my work we do have the notion of template tests. These tests are parameterized to require the specific data for the case we want to test. We then use metaprogramming to dynamically create the real tests by applying the parameters for each case to the template. This has the benefit of being deterministic, and runs as part of our automated test suite.

My TDD Practice

We do Acceptance Test Driven Development at my work. Most of our tests happen to be close to full stack functional tests. The reason is we found it was more valuable to test and assure the behavior of user driven actions. We use techniques like dynamic test generation from parameterized tests to provide us more coverage with a minimum of work. We do this for ASCII vs UTF8, API conventions, and well known variant tests.

Answer 5

The topic of Fuzzing has come up, which I think is a great approach to this problem (in general). This raises the questions: Does Fuzzing fit into TDD? Where in the TDD process does fuzzing fit?

I believe that it might fit quite well! There are fuzzers like american fuzzy lop that can be scripted and adapt themselves to modifications in the I/O format on their own. In this particular case, you could integrate it with Travis CI, store the input test cases you used and run regression testing against those.

I might extend this answer if you come up with any questions for details in the comments.