DNS Sinkhole with Sourcefire

There is this nice feature with Cisco Firepower called DNS Intelligence. This feature allows us  to have a huge database containing known bad domain names and utilize that database to drop connections to IPs represented by those names. We can have these names in form of some feed provided by Cisco or some other vendor, for free or as a payed service, or it could be created by us. What ever case may be, the point here is the same: we want to drop connections based on the result of the DNS query. One good example would be C2 or Command-and-Control connections. If one of our PCs caught some malware and that malware is trying to call home using known bad DNS, we can detect and prevent it. More on how DNS intelligence works can be found here.

Let’s see one typical scenario. The usual  query looks like this:

So, client asks for the IP of a given name (1), if the name is not malicious, the SFR passes the query to the public DNS servers  (2). The chosen DNS server returns the answer (3) which SFR passes on to the client (4). Now client connects to the returned IP address, with the HTTP(s) or any other protocol.

If a request contains a malicious domain, then the SFR could return a sinkhole IP address, if instructed to do so, of course:

The steps are almost identical. The only difference is that the SFR recognizes that the requested DNS name is malicious and returns the sinkhole IP address instead of real IP. Now the client connects to the sinkhole address of 5.x.y.133 and can be easily tracked and identified as infected. The site in question is by no means malicious, but rather an example for testing purposes.

Now comes a second scenario:

The query flow is similar, but instead of asking public DNS server directly, the infected PC is asking our private DNS servers for address (1) and our DNS server in turn asks public DNS servers (2). If the name is not malicious, SFR will pass the request to the Internet (3) and the resolved IP address will be returned to the client (4), (5) and (6).

If the requested name is malicious, then in step (5) the SFR will return the IP address of the sinkhole object and our private DNS server will just pass this info to the client (6):

Now the client connects to the sinkhole IP and we got it logged on the FMC.

There are two issues with the second scenario: because the SFR is seeing a malicious request coming from our DNS server (2), it will mark DNS server as being possessed by a malware, which clearly is not the case. So, two not so good things are happening here: first, our DNS server (and probably a domain controller) is marked with “Indication of Compromise” flag, and second, there is no way of telling which PC is actually infected. This is where DNS Sinkhole action comes into play. It fixes the second issue, so we can track infected clients. I am afraid that the DNS servers will always be marked with an indication of compromise flag, but we can treat these events as a false positive, given that we know what we are doing.

The sinkhole is nothing more than an IP address that the SFR will return for DNS queries made either by clients directly or via internal DNS server. This can be fake, non used address or it can be an IP address of real server. The important thing is that the address *must* be “outside” of the network, or should I say in a place in our organization so when the query that client sends to this address actually goes *through* the SFR. This is important because this way the SFR can catch follow-ups connections made by our clients after DNS queries. I mean real HTTP/HTTPs or other connection types. If the IP address is bogus, then we can filter out all events on SFR with the destination IP of sinkhole object and associated source IPs are actually our infected PCs. If the IP address is real, that is we have a server on that IP, we can have more data collected on that server to do deeper investigation. Makes sense?

We already know how DNS policy works, how we configure it and where we attach it. Now we are going to alter our policy in this way: we will create a list of DNS names we want to sinkhole. Then we will create a sinkhole object. Finally we will create a DNS policy rule that will return a sinkhole object address for any query sent for names from the list we created.

First, let’s create our list. “Objects->Object Management->Security Intelligence->DNS Lists and Feeds”. We click “Add DNS Lists and Feeds“. We give this list a name, select type and browse to the file. This file contains DNS names we would like to sinkhole, one name in one line. Never mind the path that is displayed that contains “fakepath“. Then we upload the file and click Save.

Now we create a sinkhole object. Like said previously, this can be fake IP or the address of a real server that will be collecting additional data that comes from our infected clients. The IP address must be routable in a way that actual connection from infected PC to this address passes through the SFR. Also, the IPv6 address is mandatory. We are not using IPv6, so it does not matter what we put here as long as it is a valid address.

Creating a sinkhole object is done via tree option “Sinkhole” under object management:

We give it a name, IPv4, IPv6 addresses, select if we want just to log or log and drop follow-up connections to the sinkhole address. Optionally we can set a type which will be logged on the FMC. This screen shot depicts IP address of This is just an example. We will actually use here an IP address of 5.x.y.133.

Now we need to modify our existing DNS policy to include a rule that will only trigger if the request is coming from specific IPs and contains a specific query. This is for testing purposes of course, because we don’t want to affect the entire organization. Once we have tested this, we can remove source IP filter and expand our list, or even include some feed.

So, our tabs should be filled like this:

And our rule should be positioned at the right place inside the DNS policy, because the rules are evaluated from top to bottom:

Now we must save changes and apply access control policy.

After policy is applied, it is time for testing. First, scenario in which clients asks public DNS servers directly:

Clearly the our SFR returned a sinkhole object. This can be verified under “Analysis->Connection->Security Intelligence Events“:

If we now browse to the wanted site we are actually going to go to the 5.x.y.133. This will be logged and if we had something listening on this IP, we could also have packets captured on that server.

So far, everything was hunky-dory. Now, if we try all this with our internal DNS servers, we get an unexpected result:

This is where all the fun begins. According to Cisco’s documentation, DNS inspection on the ASA firewall can interfere with the normal sinkhole operations. So, they recommend to turn this feature off:

asa(config)# policy-map global_policy
asa(config-pmap)# class inspection_default
asa(config-pmap-c)# no inspect dns preset_dns_map

This did not help! So, some smart people from Cisco suggested that yet another feature should be turned off – “DNS Guard“. So, plain and simple:

asa(config)# no dns-guard


No luck here as well!

Before we move on, I should stress out that one should read and understand about turning these features off because that way overall security may actually be lowered. I did not investigate upon this, but we should have this in mind. Anyhow, turning these features off did not help. Finally, there is documented bug for Cisco Sourcefire 6.0.1 and 6.1.0 – “DNS Sinkhole does not work with EDNS” (bugID: CSCvb99851). It says that Windows 2012R2 DNS server can cause issues with Sourcefire by using some extended attributes in queries it sends and those attributes are tough to be processed by Sourcefire. This attribute or extension is marked as EDNS0 and allows Windows DNS server to send UDP packets larger than 512 bytes. I don’t think this is the case, because I have turned off DNS packet size checking policy on the ASA. I think this is something only Microsoft DNS servers understand or use. Anyhow, we just turned off features on ASA that blocks larger DNS queries, and our SFR version is 6.2.0, so this should really not be an issue, but let’s give it a try…

The feature is turned off on Windows 2012 R2 DNS server by running this command:

dnscmd /config /enableednsprobes 0

This modification should disable the extension in question. The success message should be displayed:

Registry property enableednsprobes successfully reset.
Command completed successfully.

Now again, please read about implications on your environment if turning this feature off. I have came across some users complaining that after some time, the changed value reverts itself back to original and again breaks our sinkhole operations.

I took another approach and changed the registry myself. On Windows 2012 R2 Server registry key actually does not exist. The branch in question is:


and key value is


So, we need to add this key with a value of zero:

After this change, we must restart our DNS service.

Let’s now try to resolve our test domain name and see if we will get a sinkhole object back:

We are using different names for testing, because our clients and DNS servers cache results, so if previous test did not come up as we expected, we could fail over and over again because the response was cached.

As we can see, we are now asking our internal DNS server and are getting a real sinkole IP address. Now it it easy to identify all PCs infected with some sort of malware. We need to check our logs on FMC by looking for a destination IP address of our sinkhole object and whatever IP addresses we find in the source column, those are infected machines:


An useful concept, actually easy to explain but somehow hard to implement. Not any more 🙂

Thanks for reading.


This entry was posted in Cisco, FirePOWER, FireSight, IPS, Security, Sourcefire and tagged , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s