Cisco Sourcefire 5.3.x to 5.4.x Upgrade

I was thinking whether or not publish this one. Upgrading FirePOWER from 5.3.x to 5.4.x is perhaps most trickier of all upgrades I have ever done. Now, wait a sec, somebody will say, upgrade the DefenseCenter and then upgrade SFR modules. How tough can that be? Well, conceptually, that’s exactly how it’s done, but we have to read a ton of papers in order to have it all done right. We have to take care of versions for both DC and SFRs, as well as ASAs, free disk space, hypervisor versions, upgrade paths and so on. What bothers me most is that in upgrade documentation, several times we can find the sentence like “If something goes wrong, please contact Support.” I don’t like that, but the upgrade needs to be done, so let’s begin.

Our current version is 5.3.1, which can be verified by going to Help->About in our Defense Center:

helpabout

Our SFR modules should be running about the same version. We verify this in the following portion of the DC: Devices->Device Management->Device->System:

sfrversion

Our task in this blog is to upgrade DC to the 5.4.1.3 and SFR modules to 5.4.0.4. These are the latest available versions at the time this blog is written. We will do an upgrade to 6.x later, as soon as that version gets available.

Here is what we are going to do:

  • Make sure that there is no any health issues raised within the DC
  • Reapply our policies
  • Transfer our 5.3.x backups to external storage, because the upgrade will delete them
  • Download required upgrade scripts and verify check sums
  • Plan the upgrade for off-peek hours
  • Transfer our upgrade files to the DC
  • Read as many documents as possible on the topic
  • Upgrade our DC to 5.4.0
  • Upgrade the DC to 5.4.1.3
  • Upgrade SFR modules to 5.4.0
  • Upgrade modules to 5.4.0.4

 

Read the release notes, don’t be lazy!

I’m not kidding! This blog is summary of the upgrade process and the upgrade process using this blog should go smoothly. After all, it has been tested in both lab and production environments. However, there are lots of stuff that can go wrong. For example version numbers, health issues, supported browsers, plugins and so on. Always have the “Plan B” prepared: what to do if something goes wrong. The upgrade process itself is not that tough, once the preparation is done as it should be.

 

Taking care of health issues
Under the Health menu, we have to make sure that there is no any issues, warnings, errors or criticals. We do not proceed until all is green.

health1

Here we have two critical events, but they are found to be a false positives. In this specific case, the DC showed unusual high memory usage on two SFR modules, but after investigation it turned out to be a bug in SFR version 5.3.1 which reported the wrong memory usage to the DC and that actually memory allocation was just fine. So, we still have all green, although we have an critical reported by DC. Again, don’t start the process until all warnings/errors/criticals are corrected.

Reapply our policies
Because the upgrade process will reboot the DC and all modules, we need to make sure that our policies are applied, so we don’t lose any changes we made. We should reapply policies after each upgrade step.

Transfer our backups to external storage
The migration process will delete all 5.3.x backups from the DC, so we have to transfer our backups to external location. Here we can read how to do a backup and transfer it to a safe external location.

Download packages and verify checksums
We have to have a valid subscription in order to download needed packages. These are actual shell scripts that will be executed on the DC or SFR modules. We need the following packages:

Sourcefire_3D_Defense_Center_S3_Upgrade-5.4.0-763.sh
Sourcefire_3D_Defense_Center_S3_Patch-5.4.1.3-55.sh
Cisco_Network_Sensor_Upgrade-5.4.0-763.sh
Cisco_Network_Sensor_Patch-5.4.0.4-55.sh

First one is a DC upgrade from 5.3.1 to 5.4.0, second one is a patch that is going to move our basic 5.4.0 version to 5.4.1.3. The third one is upgrade for SFR module from 5.3.1 to 5.4.0, and finally, last one is going to move SFR version from 5.4.0 to 5.4.0.4.We need to verify sums of each of them, because we don’t want to break our upgrade process because we had an error while downloading the file. There are many tools available for calculating hashes. For windows we can use FCIV tool, for example. This tools have to be downloaded and installed. On Linux, there are tools already installed. From Cisco site, we copy hash values, for example SHA512 and then, once the file is downloaded, we calculate the hash ourselves. For example:

$
$ sha512sum Sourcefire_3D_Defense_Center_S3_Upgrade-5.4.0-763.sh
$

We verify the calculated sum against one we copied from Cisco site. If we have a match, we can proceed. We do this for all packages.

Transfer files to DC
We are going to upload all four files at once. Under System->Updates, we click “Upload Update” and browse for the file:
updates

After all files are updated, we can see them listed in the DC:

updates4

 

Upgrade the Defense Center
This is where the action begins. Before we go to the 5.4.1.3, we need to go to 5.4.0 first. In our scenario, we have virtual DC. There are no special requirements as far as virtualization is concerned. Supported platforms are VMware ESXi 5.1 and 5.5. Beside this, on the DC, we have to have 300MB free disk space for / partition and at least 5.5GB free disk space for /Volume partition. This requirement is for upgrading the DC. For SFR modules upgrade, we need 100MB of free disk space on SFR for / partition and 3.5GB free space for /Volume partition. If we are upgrading SFR modules from DC instead of CLI, we need additional 1GB of free space on /Volume partition on the DC. We can verify our DC disk space status by going to Dashboards->Summary Dashboard->Status->Disk Usage:

diskusage

Here is one thing for consideration: although this is virtual platform, I didn’t find anything regarding making snapshot of the DC virtual machine before doing upgrade, just in case we are forced to revert back if the upgrade fails. Perhaps this can be done, but regarding failed DC upgrades, the release notes says:

If the update fails for any reason, the page displays an error message indicating the time and date of the failure, which script was running when the update failed, and instructions on how to contact Support. Do not restart the update.Caution: If you encounter any other issue with the update (for example, if a manual refresh of the Update Status page shows no progress for several minutes), do not restart the update. Instead, contact Support

So, if there is no anything about recovering from failed update using snapshots, I would not try using them.

To initiate the upgrade, we go to System->Updates and click the update icon next to the update we are about to start:

startupgrade

The upgrade of the DC will reboot the DC. Before that, at some stage, we will be logged out and when we log back in, we can track the upgrade process. It is important to note that the upgrade and restart of the DC will not cause interruption in the SFR modules traffic flow. We just cannot manage devices while the DC is upgrading. Once the upgrade starts, it will take some time to complete. Cisco does not provide any estimates, because the speed of the process depends on the hardware platform the DC runs on. Before we actually click that button, we have to have in mind that we cannot roll back from 5.4.x to 5.3.x, because the upgrade process deletes all uninstaller scripts. If we want to go back, we have to (yet again) contact the Support. Once the upgrade starts, we can monitor but not interrupt it! Unless we power off virtual machine, which is very, very bad idea! Let’s click the button…

When we do so, we select the DC we want to upgrade and click Install:

startupgradedc1

When the upgrade process begins, for a while we can see the process under running tasks:

startupgradedc5

At some point, the browser session will end and we have to log back in. When we do, we can track on the update status:

updates_afterlogin

By clicking “show log for current script” we can see a log for the upgrade process:

updates_afterlogin2

As we can see, the process really takes some time. So, let’s grab a cup of coffee …

Now, that’s what I’m talking about! Almost two hours:

dc_upgrade_completed

Now the DC will reboot. We should clear our browser cache and restart web session. When we log into our new DC, we need to accept the EULA. After that, let’s verify our DC version. The Help->About produces the following output:

upgradedto5.4.0

Now we may need to apply new intrusion rules and install latest VDB and GeoDB. We may not be required to do so if these rules are up to date. But if we do need, this is done under System->Updates->Rule Updates.

The VDB is updated from System->Updates->Product Updates.

The GeoDB is updates from System->Updates->Geolocation Updates.

Finally, we need to reapply our policies to SFR devices. We already know how to do that.

Now we need to verify the upgrade process. Check the versions, health status, traffic logging, communication to SFR modules and so on…

This completes our first major task. We have upgraded the DC to 5.4.0 version. Before we proceed with the SFR modules upgrade, we are going to repeat the whole DC upgrade process, but this time, we are going from 5.4.0 to 5.4.1.3. The process is almost the same, but instead of selecting the upgrade file, for example

Sourcefire_3D_Defense_Center_S3_Upgrade-5.4.0-763.sh

we should chose the patch file, for example

Sourcefire_3D_Defense_Center_S3_Patch-5.4.1.3-55.sh

This file is listed as a patch within the DC, with the appropriate version number:

11-9-2015 1-02-15 PM

 

Upgrade SFR modules

At this point we have our DC at 5.4.1.3 version, but our modules are 5.3.1. We need to upgrade modules to 5.4.0 first, and then to the 5.4.0.4. In order to upgrade our modules to 5.4.0.4, the ASA software must be at least 9.3.1. Disk space requirements for module upgrades can be found in the DC upgrade section. Again, don’t be lazy and read the release notes for versions that are about to be installed!

We have to have in mind the fact that the during the upgrade, SFR modules will reboot. This can cause a traffic flow interruption. With the ASA software modules, we have two options when sending the traffic to the module:

  1. sfr fail-open
  2. sfr fail-close

The “sfr fail-open” allows traffic to flow even if the module is down for some reason, such as an upgrade. With this option, if the module’s status is up, then the traffic is sent to it, and if the module is in different state, then the traffic won’t be sent to the module, but rather go through the ASA without inspection. With “sfr fail-close” the traffic will be dropped if the module is down or rebooted.

Above mentioned applies to the data plane, which means the traffic itself. The module has also the management plane. If we remember when we deployed these software modules, they share Management interface with the physical ASA box. When the module reboots, we will not be able to access the module for the sake of management or monitoring.

This is what we are about to do. First, we will monitor the simple ICMP echo request going through the ASA/SFR while upgrading the module from 5.3.1 to 5.4.0 and with “sfr fail-open“. Then we will do the upgrade from 5.4.0 to 5.4.0.4 with the “sfr fail-close” and compare results. In the first case, we lost only two pings, perhaps because a CPU spiked, or something like that. I believe that no sessions were dropped, but I would not count on that in the production environment. In the second case, the traffic was being dropped the whole time the module was upgrading.

We trigger the update process the same way we did with the DC, by selecting appropriate update, clicking install icon and select desired module:

11-10-2015 12-05-19 PM

We have option to select the module we want to upgrade:

11-10-2015 12-57-39 PM

It would be nice if we had at least one test module, so we can do the upgrade on that one. After we confirm that the upgrade was successful, we can then apply the upgrade to some or all other modules.

We have to confirm the action:

11-10-2015 1-44-44 PM

Now we wait and watch the progress:

11-10-2015 1-03-14 PM

11-10-2015 1-27-54 PM

Success!

During the module upgrade, no activities should be performed on the DC except for monitoring the upgrade process.

Fine. We have successfully upgraded our test module and now it’s time to go with the production upgrade. If we have a single ASA/SFR, the process is exactly the same. As we saw, we missed only two pings. I suppose that all connection won’t suffer if we are in fail-open mode. But, Cisco says that we can expect traffic interruption, so we may need to plan for the downtime. The estimate SFR module upgrade is about 45 minutes, so we can expect the downtime to last close to that.

If we had an Active/Standby fail over (most probably even Active/Active, but I did not try that) then we should not have any downtime. With the A/S failover, the SFR module is treated as any other interface. This means that we should first upgrade the module inside the standby unit. During the module upgrade, active unit will pass the traffic, as usual. After the upgrade is successful, we initiate manual fail over, so the standby unit becomes active and starts passing the traffic. Then, we upgrade the other module. After this upgrade is done, we can fail over back to the previously active unit. This step is optional, though.

Once we have upgraded our modules to 5.4.0, we repeat the same steps in order to do the upgrade to the 5.4.0.4 version.

Of course, we should not forget to reapply our policies after each upgrade!

That’s all for now. Until the 6.0 version, stay safe!

 

Thanks for reading.

 

Advertisements
This entry was posted in FirePOWER, IPS, Security, Sourcefire and tagged , , , , . Bookmark the permalink.

24 Responses to Cisco Sourcefire 5.3.x to 5.4.x Upgrade

  1. Perfect timing as I am applying the 5.4.0.4 patch tonight. I am entirely new to this so seeing this post, even though it goes through the 5.3 to 5.4 process that doesn’t apply to me, reassures my nerves a little.

    Thanks and love your posts!

  2. Boyan Kurtev says:

    Version 6 has just been released ! 🙂 Looking forward for exploring SSL inspection feature !

  3. deadbeef says:

    I think as long as Cisco doesn’t make 9.4.x “star labeled” ASA OS and FirePOWER/FireSIGHT gets at least 6.1.x i won’t touch 6.x release. Its probably plagued with bugs.

  4. Pingback: Upgrade Cisco Sourcefire to 6.0 | popravak

  5. Zach says:

    Excellent write-up on this upgrade process. I went through this several weeks ago; I had to end up opening a support case due to issues with our ASAs; because of a Cisco bug. I did read the release notes very carefully; but some undisclosed bugs (at the time) led to problems where the ASA rebooted and dropped traffic on a production perimeter interface (not good to say the least). Our ASAs are running OS version 9.3.3. Also, a word of caution; after upgrading our SFR modules to 5.4.0.3 and enabling the “Inspect Archive” advanced option on the file policy this caused the Snort engine to lock up and to silently drop all traffic across the ASA due to bug CSCut39253. The Defense Center 1500s are running OS version 5.4.1.1 right now. The worst part of this bug is even if your SFR modules are set to the “sfr fail-open” option or you are running it in an IDS mode of “monitor-only” the traffic across the ASA can still drop silently even with it running a mirror of the network traffic stream.
    I saw that on November 11th OS version 6.0 has been released and that you have just written about that. I am a bit wary of upgrading to this; especially after upgrading to 5.4 and trying to use the newly enabled “Inspect Archive” feature.

  6. Lovleen says:

    hi,
    can i not just install the 6.0 on the sfr module on ASA.
    Shall I must upgrade the asa 5525-x sfr module from existing 5.3 to 5.4 and then to 6.0 etc?
    Thanks

    • Sasa says:

      If this is a clean install, then you can start with 6.0. Of course if the requirements are met, such as the ASA version, ammount of RAM for DC etc. If you already have 5.3 then you must follow these steps or wipe the 5.3 and start over, which you probably don’t want.

  7. Lovleen says:

    Few Questions:
    We are building this from scratch, ASA 5525-X is in production at the moment. SFR is not being used. It is on 5.3.1.x.
    1. Can I directly load 6.0.0.x .img and .pkg file to start fresh on the ASA sfr module?
    2. As we running ASA in active/standby, does the sfr need to be setup similarly or will I be adding both primary and secondary sfr to the FireSight-MC server and do the IPS that way.
    Many thanks

    • Sasa says:

      If you don’ use 5.3 in any way, you can load 6.0 directly, because during the install you will completely wipe out the old version. Take care of DC version, though.
      As for A/S you add both modules to DC and make sure you apply the same policies to both modules. The modules should use separate names and IP addresses. The ASA failover will see these modules just as ordinary interface and the standby unit will take over if the primary ASA fails. You can see this by issuing ‘show failover’ command.

  8. Dominique says:

    Hi everybody, i’m in the shit. 😦
    My DC3000 currently in version 5.3.1 refuse to upgrade to v 5.4. I got a error message at 70% of upgrade process. “Error installation Failed – Fatal error : Error running script 800_post/700_split_ips_advanced_settings.pl”
    When i read the script, i can see this :
    WARNING: Could not find an owner for layer Balanced Security and Connectivity
    lockObjects: Unable to get all locks!
    Unable to commit the policy: Another user is committing changes to a required policy or layer
    Please try again in a few minutes at /usr/local/sf/lib/perl/5.10.1/SF/Intrusion/Policy.pm line 603.

    Thanks all for your help.

  9. Pingback: Upgrade Cisco Sourcefire to 6.2.0 | popravak

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s