Very, very strange ASA IPSec VPN issue with a Perl script workaround

For a long period of time we, in the company I worked for, were having this strange issue with L2L IPSec to one of our partners. The issue manifests itself in a very strange way: every some number of rekeys the tunnel gets stuck and get rebuilt half way: the SA for sending direction is built ok and ASA is encrypting and sending traffic, but the other SA for receiving direction is NOT built and therefore NOT decrypting and receiving traffic.

So, in the middle of the night (most often, it’s Murphy what can I tell) we get a call that the VPN is not working.

On the ASA, this is how it looks like:

Bytes Tx     : 338522                 Bytes Rx     : 0

after issuing show vpn-session.

You can see that the Rx is zero, while Tx is a non zero value. It’s always like this.

So, we contacted Cisco TAC two times, gave them configs, logs, … and they found nothing. We have changed the border router (which occasionally was making troubles), changed the software version, changed the software version on ASA cluster, even replaced 5510s with 5520s which we had in reserve. Still the same! This became very frustrating. Beside those late night calls, we were losing money.

Cisco did suggest that the other party should get involved, but they refused to participate in this by simply ignoring our mails and calls.

Fortunately, we came across this funny workaround: users call, we connect to ASA and manually clear the VPN session by the means of executing vpn-session logoff command.

Now we had two choices: losing our minds or fixing this. The only thing that crossed my mind was to find some Perl script (I have heard of such scripts somewhere) and try to modify it to match our needs. I thought: if the script can read and save ASAs config, then perhaps it can execute other commands, such as show vpn-session  and  vpn-session logoff which we need to automate the whole process.

So I found this script that executes commands and stores the results in text file. You can find the original script at http://www.scripthat.com/tag/cisco-asa/

Of course, I had to make some changes, and I attach my final script. I’m not a Perl script expert, so this could be made better… Basically, this script connects to ASA box, executes show vpn-session l2l filter name x.y.z.w and stores the result in the file. After that, it parses the file line by line and seeks for “Bytes Rx       : 0” and if it finds it kills the session by executing another command vpn-session logoff tunnel-group x.y.z.w noconfirm. After that, it sends a mail that notifies about this. I have put this script on a Linux box in the crontab to execute every minute. Now, finally we can get some sleep and try to find the real solution for this.

By the way, this script should be executed manually first time so the Linux can store ASAs SSH public key in appropriate location. And not to forget: this VPN session was working perfectly well for years before this started all by its own, with no intervention, at least from our side.

Finally, this is the script:

#!/usr/bin/perl -w
use Expect;
use strict;
use Net::SMTP;

sub grcipukli {
my $smtp = Net::SMTP->new (“mailserver.oof.rab”);
$smtp->mail (“GRCI\.PUKLI\@oof\.rab”);
$smtp->to (“sasa\.popravak\@oof\.rab”);
$smtp->data ();
$smtp->datasend (“GRCI PUKLI!”);
$smtp->dataend ();
$smtp->quit ();
}

my $lakmus = “green”;
my $username = “someuser”;
my $password = “somepass”;
my $enable = “someenablepass”;
my $host = “1.2.3.4”;
#my $host2 = “5.6.7.8”;
my @hosts;
my $backupdir = “/root/ASA/logs”;
my $timeout = 10;
my @output;
my $match = ‘Bytes Rx     : 0’;
push @hosts, “$host”;

foreach $host (@hosts) {
my $filename = “GRCI.VPN”;
my $filepath = “$backupdir/$filename”;
my $command = Expect->spawn(“ssh $username\@$host”);
$command->expect($timeout, -re => “someuser\@1.2.3.4’s password:”) or die(“Failed to get password prompt”);
print $command “$password\r”;
sleep 2;
print $command “enable\r”;
$command->expect($timeout, -re => “Password:”) or die(“Did not get a password prompt\n”);
print $command “$enable\r”;
print $command “terminal pager 0\r”;
unlink ($filepath);
$command->log_file(“$filepath”);
print $command “show vpn-sessiondb l2l filter name 4.2.2.2\r”;
my $redo = 1;
while($redo)
{ $command->clear_accum();
$command->expect(1, [ qr/More/ => sub { my $comand = shift; print $command “\r”; exp_continue; } ], [ qr/#/ => sub { my $exp = shift; $redo = 0; exp_continue; } ], );
}

open (FILE, “</root/ASA/logs/GRCI.VPN”);
while (<FILE>) {
my ($line) = $_;
chomp ($line);
if ( $line =~ /Bytes Rx     : 0/ ) {
$lakmus = ‘red’;
}
}

close (FILE);

if ( $lakmus eq ‘red’ ) {
print “\n\n\nE, JB GA!\n\n\n”;
print $command “vpn-sessiondb logoff tunnel-group 4.2.2.2 noconfirm\r”;
&grcipukli;
} else {
print “\n\n\nSVE U REDU!\n\n\n”;
}

print $command “exit\r”;
$command->soft_close();
$command->log_file(undef);
}

Hope you will NOT be in the possiotion tu use it 

Advertisements
This entry was posted in ASA, Cisco, VPN and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s