Too many DNS lookups in an SPF record

Question 1

So, I've never had to do this before, but based on the article you sent over, this is what I came up with.

We started with:

v=spf1 a mx include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all

We get 10 total lookups before we throw the Too many DNS lookups error:

  2 (Initial TXT & SPF Lookups)
  2 (a & mx Lookups)
  1 (_spf.google.com)
  1 (servers.mcsv.net)
 +1 (sendgrid.net)
 -----------------
  7 Lookups

So without even following the included SPF records, we have 7 lookups.

Now, let's dive a level deeper.

1. _spf.google.com

The google SPF record evaluates to:

v=spf1 include:_netblocks.google.com include:_netblocks6.google.com ?all

Each of which resolve to the following values:

# _netblocks.google.com
v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all

# _netblocks6.google.com
v=spf1 ip6:2607:f8b0:4000::/36 ip6:2a00:1450:4000::/36 ?all

So google gives us 2 more lookups, bringing the total up to 9 Lookups.

2. servers.mcsv.net

Mailchimp is a bit of a doosey because it adds a whole 3 extra lookups:

v=spf1 include:spf1.mcsv.net include:spf2.mcsv.net include:spf.mandrillapp.com ?all

I would imagine that depending on what you're sending through Mailchimp, you might be able to remove one or two of these records (but you'll have to evaluated that yourself).

Anyway, those resolve to the following:

# spf1.mcsv.net
v=spf1 ip4:207.97.237.194/31 ip4:207.97.238.88/29 ip4:207.97.240.168/29 ip4:69.20.10.80/29 ip4:69.20.41.72/27 ip4:74.205.22.1/27 ip4:69.20.90.0/26 ?all

# spf2.mcsv.net
v=spf1 ip4:204.232.163.0/24 ip4:72.26.195.64/27 ip4:74.63.47.96/27 ip4:173.231.138.192/27 ip4:173.231.139.0/24 ip4:173.231.176.0/20 ip4:205.201.128.0/24 ?all

# spf.mandrillapp.com
v=spf1 ip4:205.201.136.0/24 ip4:205.201.137.0/24 ?all

This brings us up to a total of 12 Lookups (Which is two over the limit already).

2. sendgrid.net

SendGrid ends up being the fewest number of additional lookups for us.

v=spf1 ip4:208.115.214.0/24 ip4:74.63.202.0/24 ip4:75.126.200.128/27 ip4:75.126.253.0/24 ip4:67.228.50.32/27 ip4:174.36.80.208/28 ip4:174.36.92.96/27 ip4:69.162.98.0/24 ip4:74.63.194.0/24 ip4:74.63.234.0/24 ip4:74.63.235.0/24 include:sendgrid.biz ~all

So the only additional lookup here is sendgrid.biz, which evaluates to:

v=spf1 ip4:208.115.235.0/24 ip4:74.63.231.0/24 ip4:74.63.247.0/24 ip4:74.63.236.0/24 ip4:208.115.239.0/24 ip4:173.193.132.0/24 ip4:173.193.133.0/24 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ~all

This brings our grand total up to 14 lookups.

So our grand total is 14 Lookups. We need to get that down to 10. I've outlined a couple of options below, you may need to use more than 1 of them to get it down.

Directly include some of the redirected spf records. Now that we know which servers the spf records redirect to, you could cut out the middleman and include them directly. Note: If any of the services end up changing their SPF records, you'll have to go through the process of updating yours manually.
Remove some of the services that you're using. Not sure what your use case is for having all of these services, but there's definitely some overlap that you might be able to use. For instance, SendGrid supports (1) transactional outgoing mail, (2) newsletter / marketing emails, and (3) incoming mail. So there may be some reducible redundancy.
Remove the MX record if it is redundant. Depending on your setup, the MX lookup can be redundant.

Hope this helps!

Question 2

Swift's answer is excellent.

A technique which is not mentioned above is to look at whether separate subdomains with their own SPF records can be used for systems that send mail via these different routes.

E.g. if the domain is example.com, have google apps send from addresses like user@gapps.example.com. Then there can be a SPF record for gapps.example.com which includes _spf.google.com, and _spf.google.com can be removed from the main example.com SPF record , which reduces lookups by 3.

Question 3

Have a look at SPF-tools* which help with reassembling the SPF record from the original one that uses includes into a static one containing merely ip4 and ip6 fields. It can be easily coupled together with locally-run DNS server or hosted DNS service using their API to keep everything in sync with the upstream includes.

*I am the author (now along with other contributors) and it is open source under Apache 2.0 license.

Question 4

We explored flattening the SPF records into IPs as well a creating subdomains. All of them seemed like a lot of work. We found a service from spfproxy.org which literally takes a couple minutes to setup. They basically mask the DNS lookups behind them using SPF macros. Not sure why more companies don't offer this.

Question 5

A few years ago I wrote hydrate-spf, a tool that looks up includes and merges the result into one giant record. As mentioned in the README, this approach isn't ideal - it removes the ability for your included domains to update their records. However, when you're bumping up against the allowed limit, it will solve the immediate problem, and can be kept somewhat maintainable through regular updates.

Question 6

The 10 lookup limit is a limit for DNS lookups. Flattening the SPF record to include less DNS lookups and substituting them for IPs (flattening) is a way to get around the limit.

You could do this manually, but then you have to update your SPF records every time one of the providers changes their IPs (which happens frequently).

The ideal solution is to use an SPF flattening service. This one is free for low volumes, or cheap for more than 500 emails/month. It regularly polls the SPF records you want to include for updated IPs.

Fraudmarc.com

Disclosure: I am not associated with this company and this is not a referral link

Question 7

This 10-DNS-lookup limit is imposed by SPF implementations to prevent DDoS attacks against the DNS infrastructure.

With DMARCLY's Safe SPF feature, you can lift the limit without rewriting your SPF record.