Stuck thread on Zope when Plone 3.3 content rule tries to send an email

Question

I'm trying to get to the bottom of a problem that's causing our Zope instance to lock up with increasing regularity. I've installed Products.signalstackLogger to try to get to the bottom of it and it appears it's happening when one of the content rules we have defined in Plone is trying to send an email.

For example:

Thread 46447504 (GET /VirtualHostBase/https/xxx.xxx.xxx:443/plone/VirtualHostRoot/news/portal_factory/News Item/news_item.2012-09-18.0763501726/edit):
  File "/home/zope/home/parts/zope2/lib/python/ZServer/PubCore/ZServerPublisher.py", line 31, in __init__
    response=b)
  File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 401, in publish_module
    environ, debug, request, response)
  File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 202, in publish_module_standard
    response = publish(request, module_name, after_list, debug=debug)
  File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 119, in publish
    request, bind=1)
  File "/home/zope/home/parts/zope2/lib/python/ZPublisher/mapply.py", line 88, in mapply
    if debug is not None: return debug(object,args,context)
  File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 42, in call_object
    result=apply(object,args) # Type s<cr> to step into published object.
  File "/home/zope/home/eggs/Plone-3.3.6-py2.4.egg/Products/CMFPlone/FactoryTool.py", line 375, in __call__
    temp_obj = tempFolder.__getitem__(id)
  File "/home/zope/home/eggs/Plone-3.3.6-py2.4.egg/Products/CMFPlone/FactoryTool.py", line 155, in __getitem__
    self.invokeFactory(id=id, type_name=type_name)
  File "/home/zope/home/eggs/Plone-3.3.6-py2.4.egg/Products/CMFPlone/PloneFolder.py", line 348, in invokeFactory
    new_id = pt.constructContent(*args, **kw)
  File "/home/zope/home/eggs/Products.CMFCore-2.1.3-py2.4.egg/Products/CMFCore/TypesTool.py", line 716, in constructContent
    ob = info.constructInstance(container, id, *args, **kw)
  File "/home/zope/home/eggs/Products.CMFCore-2.1.3-py2.4.egg/Products/CMFCore/TypesTool.py", line 276, in constructInstance
    ob = self._constructInstance(container, id, *args, **kw)
  File "/home/zope/home/eggs/Products.CMFCore-2.1.3-py2.4.egg/Products/CMFCore/TypesTool.py", line 450, in _constructInstance
    newid = m(id, *args, **kw)
  File "<string>", line 7, in addATNewsItem
  File "/home/zope/home/parts/zope2/lib/python/OFS/ObjectManager.py", line 349, in _setObject
    notifyContainerModified(self)
  File "/home/zope/home/parts/zope2/lib/python/zope/app/container/contained.py", line 365, in notifyContainerModified
    notify(ContainerModifiedEvent(object, *descriptions))
  File "/home/zope/home/parts/zope2/lib/python/zope/event/__init__.py", line 23, in notify
    subscriber(event)
  File "/home/zope/home/parts/zope2/lib/python/zope/component/event.py", line 26, in dispatch
    for ignored in zope.component.subscribers(event, None):
  File "/home/zope/home/parts/zope2/lib/python/zope/component/_api.py", line 130, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/home/zope/home/parts/zope2/lib/python/zope/component/registry.py", line 290, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/home/zope/home/parts/zope2/lib/python/zope/interface/adapter.py", line 535, in subscribers
    subscription(*objects)
  File "/home/zope/home/eggs/plone.app.contentrules-1.1.7-py2.4.egg/plone/app/contentrules/handlers.py", line 174, in modified
    execute(aq_parent(aq_inner(event.object)), event)
  File "/home/zope/home/eggs/plone.app.contentrules-1.1.7-py2.4.egg/plone/app/contentrules/handlers.py", line 97, in execute
    executor(event, bubbled=True, rule_filter=rule_filter)
  File "/home/zope/home/eggs/plone.contentrules-1.1.0-py2.4.egg/plone/contentrules/engine/executor.py", line 27, in __call__
    executable()
  File "/home/zope/home/eggs/plone.contentrules-1.1.0-py2.4.egg/plone/contentrules/rule/rule.py", line 48, in __call__
    if not executable():
  File "/home/zope/home/eggs/plone.app.contentrules-1.1.7-py2.4.egg/plone/app/contentrules/actions/mail.py", line 103, in __call__
    From=source)
  File "/home/zope/home/eggs/Products.SecureMailHost-1.1.2-py2.4.egg/Products/SecureMailHost/SecureMailHost.py", line 246, in secureSend
    return self._send(mfrom, all_recipients, msg, debug=debug)
  File "/home/zope/home/eggs/Products.SecureMailHost-1.1.2-py2.4.egg/Products/SecureMailHost/SecureMailHost.py", line 276, in _send
    mail.send()
  File "/home/zope/home/eggs/Products.SecureMailHost-1.1.2-py2.4.egg/Products/SecureMailHost/mail.py", line 102, in send
    smtpserver = smtplib.SMTP(self.host, self.port)
  File "/usr/local/lib/python2.4/smtplib.py", line 244, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/local/lib/python2.4/smtplib.py", line 311, in connect
    (code, msg) = self.getreply()
  File "/usr/local/lib/python2.4/smtplib.py", line 352, in getreply
    line = self.file.readline()
  File "/usr/local/lib/python2.4/socket.py", line 332, in readline
    data = self._sock.recv(self._rbufsize)

These notifications generally work without issue (many emails are sent per day) and have been configured well over 12 months with the problem only arising in the last 2 months. The only major change that proceeded the problem was switching from Apache load balancer to Haproxy

We're using:

• Linux
• Local qmail server (localhost:25)
• Python 2.4.6
• Plone 3.3.6

Answer 1

This seems to be the result of our qmail server hitting its external connection limit. We have this configured to 30 and one IP (presumably a spambot) was taking up 20 of these.

Our Zope instances send email via qmail on the same virtual server but that is also our externally facing mail server. Once the connection limit is reached it seems that connection attempts from smtplib will wait with no timeout.

For now I've turned off the offending content rules. Longer term we may patch ucspi-tcp to limit connections by IP and switch to using a separate mail server for local and external emails.