当断言失败时，继续在Python的UNDIST
https://stackoverflow.com/questions/4732827
-
12-10-2019 - |
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
Russian题
编辑：切换到一个更好的示例，并澄清为什么这是一个真正的问题。
我想在Python中编写单元测试，该测试在断言失败时继续执行，以便在一次测试中看到多个故障。例如：
class Car(object):
def __init__(self, make, model):
self.make = make
self.model = make # Copy and paste error: should be model.
self.has_seats = True
self.wheel_count = 3 # Typo: should be 4.
class CarTest(unittest.TestCase):
def test_init(self):
make = "Ford"
model = "Model T"
car = Car(make=make, model=model)
self.assertEqual(car.make, make)
self.assertEqual(car.model, model) # Failure!
self.assertTrue(car.has_seats)
self.assertEqual(car.wheel_count, 4) # Failure!
在这里，测试的目的是确保汽车的 __init__ 正确设置其字段。我可以将其分解为四种方法（这通常是一个好主意），但是在这种情况下，我认为将其保留为一种测试单个概念的单个方法（“对象已正确初始化”）。
如果我们假设在这里最好不要分解该方法，那么我有一个新问题：我无法一次看到所有错误。当我修复 model 错误并重新进行测试，然后 wheel_count 出现错误。当我第一次运行测试时，它会节省我的时间来查看这两个错误。
为了进行比较，Google的C ++单元测试框架 区分 在非致命之间 EXPECT_* 断言和致命 ASSERT_* 断言：
断言成对测试相同的东西，但对当前功能产生不同的影响。 Assert_*版本失败时会产生致命的失败，并流产当前功能。 Expect_*版本会生成非致命故障，不会中止当前功能。通常，Expect_*是首选，因为它们在测试中允许报告多个失败。但是，如果在有问题的断言失败时继续进行，则应使用assert_*。
有办法获得 EXPECT_*- 像Python的行为一样 unittest？如果没有 unittest, ，那么还有另一个支持此行为的Python单元测试框架吗？
顺便说一句，我对非致命主张可能受益多少现实生活测试感到好奇，所以我看了一些 代码示例 （编辑2014-08-19使用搜索码代替Google代码搜索，RIP）。在第一页的10个随机选择结果中，所有测试都在相同的测试方法中产生了多个独立断言。所有人都将从非致命主张中受益。
解决方案
您可能想做的就是派生 unittest.TestCase 因为那是当断言失败时投掷的课程。您将不得不重新构造您的 TestCase 不要投掷（也许要保留失败列表）。重新构造的内容可能会导致您必须解决的其他问题。例如，您最终可能需要得出 TestSuite 改变对您所做的更改的支持 TestCase.
其他提示
具有非致命主张的另一种方法是捕获断言异常并将异常存储在列表中。然后断言该列表是拆解的一部分。
import unittest
class Car(object):
def __init__(self, make, model):
self.make = make
self.model = make # Copy and paste error: should be model.
self.has_seats = True
self.wheel_count = 3 # Typo: should be 4.
class CarTest(unittest.TestCase):
def setUp(self):
self.verificationErrors = []
def tearDown(self):
self.assertEqual([], self.verificationErrors)
def test_init(self):
make = "Ford"
model = "Model T"
car = Car(make=make, model=model)
try: self.assertEqual(car.make, make)
except AssertionError, e: self.verificationErrors.append(str(e))
try: self.assertEqual(car.model, model) # Failure!
except AssertionError, e: self.verificationErrors.append(str(e))
try: self.assertTrue(car.has_seats)
except AssertionError, e: self.verificationErrors.append(str(e))
try: self.assertEqual(car.wheel_count, 4) # Failure!
except AssertionError, e: self.verificationErrors.append(str(e))
if __name__ == "__main__":
unittest.main()
一个选项是立即声称作为元组。
例如：
class CarTest(unittest.TestCase):
def test_init(self):
make = "Ford"
model = "Model T"
car = Car(make=make, model=model)
self.assertEqual(
(car.make, car.model, car.has_seats, car.wheel_count),
(make, model, True, 4))
该测试的输出将是：
======================================================================
FAIL: test_init (test.CarTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\temp\py_mult_assert\test.py", line 17, in test_init
(make, model, True, 4))
AssertionError: Tuples differ: ('Ford', 'Ford', True, 3) != ('Ford', 'Model T', True, 4)
First differing element 1:
Ford
Model T
- ('Ford', 'Ford', True, 3)
? ^ - ^
+ ('Ford', 'Model T', True, 4)
? ^ ++++ ^
这表明模型和车轮计数都是不正确的。
在单个单位测试中，它被认为是具有多个断言的反模式。单位测试预计只能测试一件事。也许您正在测试太多。考虑将此测试分为多个测试。这样，您可以正确命名每个测试。
但是，有时可以同时检查多件事。例如，当您主张同一对象的属性时。在这种情况下，您实际上声称该对象是否正确。做到这一点的方法是编写一个自定义的助手方法，该方法知道如何在该对象上断言。您可以以这样的方式编写该方法，以显示所有失败的属性，例如，当断言失败时，显示了预期对象的完整状态和实际对象的完整状态。
用单独的方法进行每个断言。
class MathTest(unittest.TestCase):
def test_addition1(self):
self.assertEqual(1 + 0, 1)
def test_addition2(self):
self.assertEqual(1 + 1, 3)
def test_addition3(self):
self.assertEqual(1 + (-1), 0)
def test_addition4(self):
self.assertEqaul(-1 + (-1), -1)
我喜欢 @Anthony-Batchelor的方法，以捕获OssertionError异常。但是，使用装饰器对这种方法有轻微的变化，也是一种报告通过/失败的测试案例的方法。
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import unittest
class UTReporter(object):
'''
The UT Report class keeps track of tests cases
that have been executed.
'''
def __init__(self):
self.testcases = []
print "init called"
def add_testcase(self, testcase):
self.testcases.append(testcase)
def display_report(self):
for tc in self.testcases:
msg = "=============================" + "\n" + \
"Name: " + tc['name'] + "\n" + \
"Description: " + str(tc['description']) + "\n" + \
"Status: " + tc['status'] + "\n"
print msg
reporter = UTReporter()
def assert_capture(*args, **kwargs):
'''
The Decorator defines the override behavior.
unit test functions decorated with this decorator, will ignore
the Unittest AssertionError. Instead they will log the test case
to the UTReporter.
'''
def assert_decorator(func):
def inner(*args, **kwargs):
tc = {}
tc['name'] = func.__name__
tc['description'] = func.__doc__
try:
func(*args, **kwargs)
tc['status'] = 'pass'
except AssertionError:
tc['status'] = 'fail'
reporter.add_testcase(tc)
return inner
return assert_decorator
class DecorateUt(unittest.TestCase):
@assert_capture()
def test_basic(self):
x = 5
self.assertEqual(x, 4)
@assert_capture()
def test_basic_2(self):
x = 4
self.assertEqual(x, 4)
def main():
#unittest.main()
suite = unittest.TestLoader().loadTestsFromTestCase(DecorateUt)
unittest.TextTestRunner(verbosity=2).run(suite)
reporter.display_report()
if __name__ == '__main__':
main()
控制台的输出：
(awsenv)$ ./decorators.py
init called
test_basic (__main__.DecorateUt) ... ok
test_basic_2 (__main__.DecorateUt) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
=============================
Name: test_basic
Description: None
Status: fail
=============================
Name: test_basic_2
Description: None
Status: pass
期望在GTEST中非常有用。这是Python的方式 要旨, ，和代码：
import sys
import unittest
class TestCase(unittest.TestCase):
def run(self, result=None):
if result is None:
self.result = self.defaultTestResult()
else:
self.result = result
return unittest.TestCase.run(self, result)
def expect(self, val, msg=None):
'''
Like TestCase.assert_, but doesn't halt the test.
'''
try:
self.assert_(val, msg)
except:
self.result.addFailure(self, sys.exc_info())
def expectEqual(self, first, second, msg=None):
try:
self.failUnlessEqual(first, second, msg)
except:
self.result.addFailure(self, sys.exc_info())
expect_equal = expectEqual
assert_equal = unittest.TestCase.assertEqual
assert_raises = unittest.TestCase.assertRaises
test_main = unittest.main
PYPI中有一个软断言包 softest 这将满足您的要求。它通过收集故障，组合异常和堆叠跟踪数据而起作用，并将其报告为通常的一部分 unittest 输出。
例如，此代码：
import softest
class ExampleTest(softest.TestCase):
def test_example(self):
# be sure to pass the assert method object, not a call to it
self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle')
# self.soft_assert(self.assertEqual('Worf', 'wharf', 'Klingon is not ship receptacle')) # will not work as desired
self.soft_assert(self.assertTrue, True)
self.soft_assert(self.assertTrue, False)
self.assert_all()
if __name__ == '__main__':
softest.main()
...生产此控制台输出：
======================================================================
FAIL: "test_example" (ExampleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\...\softest_test.py", line 14, in test_example
self.assert_all()
File "C:\...\softest\case.py", line 138, in assert_all
self.fail(''.join(failure_output))
AssertionError: ++++ soft assert failure details follow below ++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The following 2 failures were found in "test_example" (ExampleTest):
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Failure 1 ("test_example" method)
+--------------------------------------------------------------------+
Traceback (most recent call last):
File "C:\...\softest_test.py", line 10, in test_example
self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle')
File "C:\...\softest\case.py", line 84, in soft_assert
assert_method(*arguments, **keywords)
File "C:\...\Python\Python36-32\lib\unittest\case.py", line 829, in assertEqual
assertion_func(first, second, msg=msg)
File "C:\...\Python\Python36-32\lib\unittest\case.py", line 1203, in assertMultiLineEqual
self.fail(self._formatMessage(msg, standardMsg))
File "C:\...\Python\Python36-32\lib\unittest\case.py", line 670, in fail
raise self.failureException(msg)
AssertionError: 'Worf' != 'wharf'
- Worf
+ wharf
: Klingon is not ship receptacle
+--------------------------------------------------------------------+
Failure 2 ("test_example" method)
+--------------------------------------------------------------------+
Traceback (most recent call last):
File "C:\...\softest_test.py", line 12, in test_example
self.soft_assert(self.assertTrue, False)
File "C:\...\softest\case.py", line 84, in soft_assert
assert_method(*arguments, **keywords)
File "C:\...\Python\Python36-32\lib\unittest\case.py", line 682, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
笔记: ：我创建并维护 softest.
我认为没有一种方法可以使用Pyunit进行此操作，也不想以这种方式看到Pyunit。
我更喜欢每个测试功能坚持一个断言（或更具体地每个测试主张一个概念）并会重写 test_addition() 作为四个独立的测试功能。这将提供有关失败的更多有用信息， 即:
.FF.
======================================================================
FAIL: test_addition_with_two_negatives (__main__.MathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_addition.py", line 10, in test_addition_with_two_negatives
self.assertEqual(-1 + (-1), -1)
AssertionError: -2 != -1
======================================================================
FAIL: test_addition_with_two_positives (__main__.MathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_addition.py", line 6, in test_addition_with_two_positives
self.assertEqual(1 + 1, 3) # Failure!
AssertionError: 2 != 3
----------------------------------------------------------------------
Ran 4 tests in 0.000s
FAILED (failures=2)
如果您认为这种方法不适合您，您可能会发现 这个答案 有帮助。
更新
看来您正在用更新的问题测试两个概念，我将它们分为两个单位测试。第一个是将参数存储在创建新对象上。这将有两个断言，一个是 make 还有一个 model. 。如果第一个失败，则显然需要修复，无论第二次通过还是失败在此关头无关。
第二个概念更值得怀疑...您正在测试是否初始化某些默认值。 为什么？在实际使用这些值的位置测试这些值会更有用（如果不使用它们，那么它们为什么在那里？）。
这两个测试都失败了，都应该。当我进行单位测试时，我对失败的兴趣要比成功的兴趣要多得多，因为这是我需要集中精力的地方。
FF
======================================================================
FAIL: test_creation_defaults (__main__.CarTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_car.py", line 25, in test_creation_defaults
self.assertEqual(self.car.wheel_count, 4) # Failure!
AssertionError: 3 != 4
======================================================================
FAIL: test_creation_parameters (__main__.CarTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_car.py", line 20, in test_creation_parameters
self.assertEqual(self.car.model, self.model) # Failure!
AssertionError: 'Ford' != 'Model T'
----------------------------------------------------------------------
Ran 2 tests in 0.000s
FAILED (failures=2)
我有一个问题 @Anthony Batchelor 答案，因为它强迫我使用 try...catch 在我的单元测试中。然后，我封装了 try...catch 逻辑替代 TestCase.assertEqual 方法。以下黑客删除 try...catch 单位测试代码的块：
import unittest
import traceback
class AssertionErrorData(object):
def __init__(self, stacktrace, message):
super(AssertionErrorData, self).__init__()
self.stacktrace = stacktrace
self.message = message
class MultipleAssertionFailures(unittest.TestCase):
def __init__(self, *args, **kwargs):
self.verificationErrors = []
super(MultipleAssertionFailures, self).__init__( *args, **kwargs )
def tearDown(self):
super(MultipleAssertionFailures, self).tearDown()
if self.verificationErrors:
index = 0
errors = []
for error in self.verificationErrors:
index += 1
errors.append( "%s\nAssertionError %s: %s" % (
error.stacktrace, index, error.message ) )
self.fail( '\n\n' + "\n".join( errors ) )
self.verificationErrors.clear()
def assertEqual(self, goal, results, msg=None):
try:
super( MultipleAssertionFailures, self ).assertEqual( goal, results, msg )
except unittest.TestCase.failureException as error:
goodtraces = self._goodStackTraces()
self.verificationErrors.append(
AssertionErrorData( "\n".join( goodtraces[:-2] ), error ) )
def _goodStackTraces(self):
"""
Get only the relevant part of stacktrace.
"""
stop = False
found = False
goodtraces = []
# stacktrace = traceback.format_exc()
# stacktrace = traceback.format_stack()
stacktrace = traceback.extract_stack()
# https://stackoverflow.com/questions/54499367/how-to-correctly-override-testcase
for stack in stacktrace:
filename = stack.filename
if found and not stop and \
not filename.find( 'lib' ) < filename.find( 'unittest' ):
stop = True
if not found and filename.find( 'lib' ) < filename.find( 'unittest' ):
found = True
if stop and found:
stackline = ' File "%s", line %s, in %s\n %s' % (
stack.filename, stack.lineno, stack.name, stack.line )
goodtraces.append( stackline )
return goodtraces
# class DummyTestCase(unittest.TestCase):
class DummyTestCase(MultipleAssertionFailures):
def setUp(self):
self.maxDiff = None
super(DummyTestCase, self).setUp()
def tearDown(self):
super(DummyTestCase, self).tearDown()
def test_function_name(self):
self.assertEqual( "var", "bar" )
self.assertEqual( "1937", "511" )
if __name__ == '__main__':
unittest.main()
结果输出：
F
======================================================================
FAIL: test_function_name (__main__.DummyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\User\Downloads\test.py", line 77, in tearDown
super(DummyTestCase, self).tearDown()
File "D:\User\Downloads\test.py", line 29, in tearDown
self.fail( '\n\n' + "\n\n".join( errors ) )
AssertionError:
File "D:\User\Downloads\test.py", line 80, in test_function_name
self.assertEqual( "var", "bar" )
AssertionError 1: 'var' != 'bar'
- var
? ^
+ bar
? ^
:
File "D:\User\Downloads\test.py", line 81, in test_function_name
self.assertEqual( "1937", "511" )
AssertionError 2: '1937' != '511'
- 1937
+ 511
:
可以发布更多的替代解决方案，以张贴在 如何正确覆盖testcase.assertequal（），产生正确的stacktrace？
我意识到这个问题是几年前从字面上提出的，但是现在（至少）有两个Python套件可以让您这样做。
一个是最柔软的： https://pypi.org/project/softest/
另一个是Python-Delayed-Assert： https://github.com/pr4bh4sh/python-delayed-assert
我也没有使用过，但是它们看起来与我非常相似。
不隶属于 StackOverflow
