Upgrade Procedure - Cisco Nexus 5000 Switch

I did this on a Nexus 5548. The process was simple - no quirky issues. Our upgrade went from 9.5.0.3 to 9.5.1.3. I had alread established IP connectivity through the management port.

Remember that the management interface is belongs to the vrf context called "management", by default.

vrf context management
  ip route 0.0.0.0/0 10.0.3.251
interface mgmt0
  ip address 10.0.1.225/22

Download the relevant files. I downloaded these two:

n5000-uk9-kickstart.5.1.3.N1.1.bin
n5000-uk9.5.1.3.N1.1.bin

Fire up your tftp server. Verify that you have enough space on your bootflash: to hold the new files:

login: admin
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

Nexus5548-2# dir
        744    Apr 20 02:08:48 2011  license_SSI15100AW7_10.lic
      49152    Apr 20 02:11:45 2011  lost+found/
       1586    Nov 10 11:47:20 2011  mts.log
   25136128    Apr 20 02:01:51 2011  n5000-uk9-kickstart.5.0.3.N1.1b.bin
  188700150    Apr 20 02:02:52 2011  n5000-uk9.5.0.3.N1.1b.bin
       4096    Jan 01 02:05:55 2009  vdc_2/
       4096    Jan 01 02:05:55 2009  vdc_3/
       4096    Jan 01 02:05:55 2009  vdc_4/

Usage for bootflash://
  331591680 bytes used
 1319313408 bytes free
 1650905088 bytes total

I had plenty of space. Now tftp the kickstart and image files into the bootflash. Use management as the vrf context!

Nexus5548-2# copy tftp://10.0.1.227/n5000-uk9-kickstart.5.1.3.N1.1.bin bootflash:n5000-uk9-kickstart.5.1.3.N1.1.bin

Enter vrf (If no input, current vrf 'default' is considered): management
Trying to connect to tftp server......
Connection to Server Established.
TFTP get operation was successful

Nexus5548-2# copy tftp://10.0.1.227/n5000-uk9.5.1.3.N1.1.bin bootflash:n5000-uk9.5.1.3.N1.1.bin

Enter vrf (If no input, current vrf 'default' is considered): management
Trying to connect to tftp server......
Connection to Server Established.
TFTP get operation was successful

Run the install command on the kickstart file.

Nexus5548-2# install all kickstart bootflash:n5000-uk9-kickstart.5.1.3.N1.1.bin system bootflash:n5000-uk9.5.1.3.N1.1.bin

Verifying image bootflash:/n5000-uk9-kickstart.5.1.3.N1.1.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n5000-uk9.5.1.3.N1.1.bin for boot variable "system".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting "system" version from image bootflash:/n5000-uk9.5.1.3.N1.1.bin.
[####################] 100% -- SUCCESS
Extracting "kickstart" version from image bootflash:/n5000-uk9-kickstart.5.1.3.N1.1.bin.
[####################] 100% -- SUCCESS
Extracting "bios" version from image bootflash:/n5000-uk9.5.1.3.N1.1.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS

Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes      disruptive         reset  Non-disruptive install not supported if L3 was enabled

Images will be upgraded according to following table:
Module       Image         Running-Version             New-Version  Upg-Required
------  ----------  ----------------------  ----------------------  ------------
     1      system            5.0(3)N1(1b)             5.1(3)N1(1)           yes
     1   kickstart            5.0(3)N1(1b)             5.1(3)N1(1)           yes
     1        bios      v3.5.0(02/03/2011)      v3.5.0(02/03/2011)            no
     1      SFP-uC                v1.0.0.0                v1.0.0.0            no
     1   power-seq                    v1.0                    v1.0            no
     3   power-seq                    v5.0                    v5.0            no
     1          uC                v1.2.0.1                v1.2.0.1            no

Switch will be reloaded for disruptive upgrade.

Do you want to continue with the installation (y/n)?  [n] y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% -- SUCCESS
Setting boot variables.
[####################] 100% -- SUCCESS
Performing configuration copy.
[####################] 100%

The switch will reboot. Login again to verify that the you are on the new version.

Nexus 5000 Switch
login: admin
Password:

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Nexus5548-2# sh ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_serie
s_home.html
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
  BIOS:      version 3.5.0
  loader:    version N/A
  kickstart: version 5.1(3)N1(1)
  system:    version 5.1(3)N1(1)
  power-seq: Module 1: version v1.0
  uC:        version v1.2.0.1
  SFP uC:    Module 1: v1.0.0.0
  BIOS compile time:       02/03/2011
  kickstart image file is: bootflash:///n5000-uk9-kickstart.5.1.3.N1.1.bin
  kickstart compile time:  12/6/2011 22:00:00 [12/07/2011 06:30:01]
  system image file is:    bootflash:///n5000-uk9.5.1.3.N1.1.bin
  system compile time:     12/6/2011 22:00:00 [12/07/2011 08:09:44]

Hardware
  cisco Nexus5548 Chassis ("O2 32X10GE/Modular Universal Platform Supervisor")
  Intel(R) Xeon(R) CPU         with 8263872 kB of memory.
  Processor Board ID JAF1515BBRA

  Device name: Nexus5548-2
  bootflash:    2007040 kB

Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 13 second(s)

Last reset at 720106 usecs after  Thu Dec 15 20:15:10 2011

  Reason: Disruptive upgrade
  System version: 5.0(3)N1(1b)
  Service:

plugin
  Core Plugin, Ethernet Plugin

Nexus5548-2#

That's it!

WOL Directed Broadcast Configuration on Cisco Layer 3 Switches

Use directed broadcast to configure the Wake On LAN (WOL) application on Cisco L3 switches. In our case we had the WOL servers connected to VLAN 1 (10.1.20.64 and .65). They are used to wake client computers on all other private VLANs. Here's the working example for our test case with clients on VLAN2:

access-list 180 permit udp host 10.1.20.64 any 

access-list 180 permit udp host 10.1.20.65 any

interface Vlan1 

ip address 10.1.0.10 255.255.0.0 

ip helper-address 10.2.255.255 

! 

interface Vlan2 

ip address 10.2.0.10 255.255.0.0 

ip directed-broadcast 180 

Note that the helper address sends the WOL broadcast to all destination subnets that are specified there (in our case to the 10.2.0.0/16 subnet) but you control what lands up on the destination VLAN with the directed broadcast command and an acl. It works if you use a supernet in the helper address command to cover all your destination networks.

Cisco ASA 5500 7.2x to 8.4x upgrade

I just completed this upgrade on two firewalls and want to share some issues I encountered. Our firewalls were using the following basic features:

• IPSEC VPN client
• Site to Site IPSEC VPN tunnel
• PAT
• Static NATs
• NAT exempt 

Firstly the image boots up and the new asdm works. You should have no concern about the ASA not booting up. In fact I had to do one that was located in a Data Center in Chicago and we were in California!

1. The known documented issue with the nat exempt command must be taken care of. The keyword unidirectional is added to the nat exempt (nat 0) rule which must be changed to bidirectional for each nat exempt command you have.
2. Since the new access rules now references the real IPs of translated addresses I found that none of these got changed on my outside access list! I had to change each ACL entry manually from the public address to its private IP address. 
3. There is a site to site IPSEC VPN tunnel built that references a translated IP of a local host in the encryption domain. The conversion process added a nat exempt rule for the encryption domain! This, of course, prevented the local host from being translated when it was trying to connect to the remote host across the tunnel. No match was found and so the tunnel never gets established! I removed that nat exempt rule to fix.

That's it! All the best in your upgrades.

Here is the Cisco reference to the 8.4x release notes.

IOS NAT and VPN

Building NAT (PAT or Static) on a Cisco router is fairly well documented and works well. However, I got tripped up again with an implementation today where the PAT would work but then the VPN tunnel would go down!

Case Study

My client has a site to site VPN tunnel to Cust2 with 6 remote networks. All local networks (192.168.0.0/21) communicate to the remote nets with their private IPs. In addition, the local networks get PAT'd to 39.100.3.68 when routed over to the Internet. There is one static NAT that only gets translated to  39.100.3.67 when communicating with 206.96.88.115 out on the Internet. In all other cases it will remain as it's real IP of 192.168.0.76.

Here is the configuration that I built. The VPN tunnel and static NAT rule using a route map to achieve our goal worked. However, as soon as I applied the PAT pool and overload translation, the tunnel went down.

crypto isakmp policy 10

 encr 3des

 authentication pre-share

 group 2

!

crypto isakmp policy 20

 encr aes

 authentication pre-share

 group 2

crypto isakmp key *** address 209.100.166.20

!

!

crypto ipsec transform-set 3DES-SHA esp-3des esp-sha-hmac

!

crypto map vpnmap 10 ipsec-isakmp

 set peer 209.100.166.20

 set transform-set 3DES-SHA

 match address cust2-vpn

interface GigabitEthernet0/0

 description to Inside

 ip address 192.168.0.8 255.255.255.0

 ip nat inside

!

interface GigabitEthernet0/1

 description Internet

 ip address 39.105.128.242 255.255.255.252

 ip nat outside

 crypto map vpnmap

!

ip nat pool natpool 39.100.3.68 39.100.3.68 netmask 255.255.255.0

ip nat inside source list ins_nat_source pool natpool overload

ip nat inside source static 192.168.0.76 39.100.3.67 route-map nat-exempt extendable

!

ip access-list extended cust2-vpn

 permit ip 192.168.0.0 0.0.7.255 192.168.16.0 0.0.0.255

 permit ip 192.168.0.0 0.0.7.255 70.185.0.0 0.0.255.255

 permit ip 192.168.0.0 0.0.7.255 200.106.176.0 0.0.7.255

 permit ip 192.168.0.0 0.0.7.255 200.106.184.0 0.0.1.255

 permit ip 192.168.0.0 0.0.7.255 206.184.246.0 0.0.0.255

 permit ip 192.168.0.0 0.0.7.255 208.134.161.0 0.0.0.255

ip access-list extended ins_nat_source

 deny   ip any 192.168.16.0 0.0.0.255

 deny   ip any 70.185.0.0 0.0.255.255

 deny   ip any 200.106.176.0 0.0.7.255

 deny   ip any 200.106.184.0 0.0.1.255

 deny   ip any 206.184.246.0 0.0.0.255

 deny   ip any 208.134.161.0 0.0.0.255

 permit ip any any

access-list 110 permit ip host 192.168.0.76 host 206.96.88.115

access-list 110 deny   ip host 192.168.0.76 any

!

!

!

route-map nat-exempt permit 1

 match ip address 110

!

The fix is to also include the Internet facing interface into the NAT exempt access rules (ins_nat_source). Even though that interface is on the 'outside', it will get subjected to translation because of the 'permit any any' rule. Here's the fixed access rule:

ip access-list extended ins_nat_source

 deny   ip any 192.168.16.0 0.0.0.255

 deny   ip any 70.185.0.0 0.0.255.255

 deny   ip any 200.106.176.0 0.0.7.255

 deny   ip any 200.106.184.0 0.0.1.255

 deny   ip any 206.184.246.0 0.0.0.255

 deny   ip any 208.134.161.0 0.0.0.255

 deny   ip host 39.105.128.242 any

 permit ip any any

FiOS Cisco PIX ARP Problem Fixed!

The Problem


When you define a static NAT on the pix you cannot get to route beyond the pix. It appears that some FIOS ONTs send arp requests with a source IP of 0.0.0.0 and the PIX will not send a proxy arp reply for the static entry but will respond itself only (me). I'm using 6.2.2 on the PIX.

In this example my PIX has an outside interface configured with 71.x.x.210. The debug shows that it responds fine.

"debug arp" on the pix shows
arp-in: rqst for me from 0.0.0.0 for 71.x.x.210, on outside
arp-in: generating reply from 71.x.x.210 0013.60ac.bf97 to 0.0.0.0 000e.861c.d518

I have the following static entry configured.

static (inside,outside) 71.x.x.216 192.n.n.1 netmask 255.255.255.255

"debug arp" on the pix shows
arp-in: request at outside from 0.0.0.0 000e.861c.d518 for 71.246.225.216 0013.60ac.bf97

So the PIX knows about the mac address of .216 but it just won't not respond! 

Solution or Work around

Upgrade to 8.0.4-32. On the PIX software download page select PIX Interim Releases  at the top part of the page...


The PIX now replies to the arp request and everything now works like it should have in the first place!

Upgrading Cisco CUE from 3.1.1 to 7.0.5

Our current version was at 3.1.1 and we needed to fix the IMAP issue with Entourage by upgrading to 3.2.x but realized I could go all the way to 7.0.x! First login to the CUE and type show software version to check current version and check that your CME version supports by consulting the compatibility matrix. My CME was at 4.1(0) - I was good to go.


Download cue-vm-k9.nm.7.0.5.zip image form here (CCO login required). Ensure you pick the one that is appropriate for your hardware. I had an NM-CUE. Extract all files into a folder onto your FTP server. I also downloaded the language pack for the US separately and place this onto the FTP server too. At some point during the installation you will be asked to select the language and for some reason the install fails when it does not see that language file on your server.


You will need an FTP server to upgrade the CUE - I did a clean install. Once sessioned into the CUE you can follow these detailed steps but here is a quick summary.

1. Backup all your files on the CUE (this is really important for a clean install). See below for instructions.
2. Verify connectivity to your FTP server.
3. Execute command on CUE: software download clean url ftp://192.168.1.1/cue-vm-k9.nm.3.2.4.pkg username xxx password yyy
4. Check status: software download status
5. Now install the software just downloaded: software install clean cue-vm-k9.nm.7.0.5.pkg
6. The system will reload after the upgrade completes.
7. Check new version: show software version
8. Restore the files from your backup server. I have the detailed steps listed below.

If you picked the wrong version of the software you will get this error message below. I downloaded the nme version during my first attempt!

WARNING:: This command will download the necessary software to 
WARNING:: complete an upgrade.  It is recommended that a backup be done 
WARNING:: before installing software. 

WARNING:: The system will briefly be brought to an offline state
WARNING:: This will terminate any active call and prevent new calls
WARNING:: from being processed.

Would you like to continue? [n] y

Downloading ftp cue-vm-k9.nme.7.0.5.pkg
Bytes downloaded :  179928 

Validating package signature ... done

Downloading ftp cue-vm-installer-k9.nme.7.0.5.prt1
Bytes downloaded :  127198 

Installer does not recognize package file format.
Attempting to upgrade installer...
starting_phase:
install-files.sh /dwnld/pkgdata/.installer_wo
add_file /dwnld/pkgdata/cue-vm-installer-k9.nme.7.0.5.prt1 1 / tgz 
Installer upgrade successful
Restarting install process...
Validating package signature ... done
Validating installed manifests .............complete.
 - Parsing package manifest files... complete.
 - Checking Package dependencies... Service Engine Bootloader can not be installed on this platform/chassis combination.

CUE Backup Procedure 

1. Config t on the CUE
2. Configure the backup server with backup server url ftp://192.168.1.1/temp/ username xxx password yyy
3. Verify with show backup server
4. Go into offline mode - offline
5. Do the back up. I normally get all the files - backup category all
6. Exit offline mode - continue

CUE Restore Procedure

1. show backup server will identify the backup ID number. Take note of this.
2. Go into offline mode - offline
3. Restore the files (my id number was 1) - restore id 1 category all
4. reload

Low Latency Queueing QoS on an MPLS Router with a Rate Limited Circuit

In a WAN architecture where routers have MPLS circuits with full bandwidth available, deploying LLQ on the WAN interface is simple as given in the example below. Here we classify the voice traffic with the ACL 140 and mark its precedence to 5 before leaving the serial interface. In addition, during times of congestion on Serial1/0, up to 1Mbps will be reserved for the voice traffic. The priority keyword ensures that this bandwidth is reserved and also ensures that the voice queue is processed before any other queue (up to 1Mbps). All other traffic will fall in a default class which follows the WFQ mechanism for QoS on the interface.

class-map match-all voice
  match access-group 110
policy-map privport-policy
  class voice
  set ip precedence 5
  priority 1000

interface Serial1/0
  ip address 10.22.13.1 255.255.255.252
  service-policy output privport-policy

access-list 110 remark Voice
access-list 110 permit ip host 10.10.1.15 host 10.20.1.15

This example is fairly straight forward and takes care of very simple applications that require voice and data transported over private WANs - whether these are private leased lines or MPLS circuits. Unfortunately, this method cannot be used when the circuit is rate limited. For example you have purchased a DS3 circuit (45Mbps) but are contracted to use just 6Mbps of that circuit. Common networking practice recommends policing the egress traffic at 6Mbps using CAR (Committed Access Rate), which rate limits the traffic to 6Mbps. Fair enough. The next logical step is to then apply LLQ policy (as depicted above) to the interface. Wrong! Recall that the rate limiting mechanism, although has some controls, is impartial as to what traffic it drops whenever the traffic exceeds the configured rate. This is a problem for any real time traffic, whether voice or video and the consequence is fairly obvious.

So how do we overcome this dilemma? The solution - use the Child Service Policy.

In our example we have a full DS3 that must be limited to 6Mbps. This implies that the total outbound traffic cannot exceed 6Mbps. In addition, prioritize voice for 1Mbps and in the absence of voice, data should be able to use available bandwidth up to 6mbps. The configuration below solves this problem.

class-map match-all voicesignal
  match access-group 120
class-map match-all voice
  match ip rtp 16384 16383

policy-map child-policy
  class voice
  priority 1000
  set precedence 5
class voicesignal
  bandwidth 100

policy-map parent-policy
class class-default
shape average 6000000
service-policy child-policy

interface Serial1/0
  ip address 10.71.28.138 255.255.255.252
  dsu bandwidth 44210
  service-policy output parent-policy

access-list 120 permit tcp any any eq 2000
access-list 120 permit tcp any eq 2000 any
access-list 120 permit udp any any eq 2000
access-list 120 permit udp any eq 2000 any

we first classified the voice packets into two parts - voice packets (uses rtp protocol) and the voice signaling packets (SCCP - Skinny Call Control Protocol). Yeah, you guessed it, we are using Cisco Call Manager! These packets are then mapped into a policy called 'child-policy' which gives priority to voice and reserves up to 1Mbps for it during times of congistion. At the same time, 100Kbps of bandwidth is reserved for the signaling packets so that other traffic cannot 'bump' these packets when congestion occurs.

The 'parent-policy' is shaped at 6Mbps and calls on the child-policy in a nested fashion. Think of the parent-policy as an interface running at 6Mbps and you are applying an LLQ policy to it just like in our first example above. The voice and data (default class) gets nicely queued up based on the policy before it hits the actual interface where the 'parent-policy' is applied.