Tuesday, April 7, 2009

Low Latency Queueing QoS on an MPLS Router with a Rate Limited Circuit

In a WAN architecture where routers have MPLS circuits with full bandwidth available, deploying LLQ on the WAN interface is simple as given in the example below. Here we classify the voice traffic with the ACL 140 and mark its precedence to 5 before leaving the serial interface. In addition, during times of congestion on Serial1/0, up to 1Mbps will be reserved for the voice traffic. The priority keyword ensures that this bandwidth is reserved and also ensures that the voice queue is processed before any other queue (up to 1Mbps). All other traffic will fall in a default class which follows the WFQ mechanism for QoS on the interface.

class-map match-all voice
  match access-group 110
policy-map privport-policy
  class voice
  set ip precedence 5
  priority 1000

interface Serial1/0
  ip address
  service-policy output privport-policy

access-list 110 remark Voice
access-list 110 permit ip host host

This example is fairly straight forward and takes care of very simple applications that require voice and data transported over private WANs - whether these are private leased lines or MPLS circuits. Unfortunately, this method cannot be used when the circuit is rate limited. For example you have purchased a DS3 circuit (45Mbps) but are contracted to use just 6Mbps of that circuit. Common networking practice recommends policing the egress traffic at 6Mbps using CAR (Committed Access Rate), which rate limits the traffic to 6Mbps. Fair enough. The next logical step is to then apply LLQ policy (as depicted above) to the interface. Wrong! Recall that the rate limiting mechanism, although has some controls, is impartial as to what traffic it drops whenever the traffic exceeds the configured rate. This is a problem for any real time traffic, whether voice or video and the consequence is fairly obvious.

So how do we overcome this dilemma? The solution - use the Child Service Policy.

In our example we have a full DS3 that must be limited to 6Mbps. This implies that the total outbound traffic cannot exceed 6Mbps. In addition, prioritize voice for 1Mbps and in the absence of voice, data should be able to use available bandwidth up to 6mbps. The configuration below solves this problem.

class-map match-all voicesignal
  match access-group 120
class-map match-all voice
  match ip rtp 16384 16383

policy-map child-policy
  class voice
  priority 1000
  set precedence 5
class voicesignal
  bandwidth 100

policy-map parent-policy
class class-default
shape average 6000000
service-policy child-policy

interface Serial1/0
  ip address
  dsu bandwidth 44210
  service-policy output parent-policy

access-list 120 permit tcp any any eq 2000
access-list 120 permit tcp any eq 2000 any
access-list 120 permit udp any any eq 2000
access-list 120 permit udp any eq 2000 any

we first classified the voice packets into two parts - voice packets (uses rtp protocol) and the voice signaling packets (SCCP - Skinny Call Control Protocol). Yeah, you guessed it, we are using Cisco Call Manager! These packets are then mapped into a policy called 'child-policy' which gives priority to voice and reserves up to 1Mbps for it during times of congistion. At the same time, 100Kbps of bandwidth is reserved for the signaling packets so that other traffic cannot 'bump' these packets when congestion occurs.

The 'parent-policy' is shaped at 6Mbps and calls on the child-policy in a nested fashion. Think of the parent-policy as an interface running at 6Mbps and you are applying an LLQ policy to it just like in our first example above. The voice and data (default class) gets nicely queued up based on the policy before it hits the actual interface where the 'parent-policy' is applied.

Thursday, April 2, 2009

VLAN Filters Not Working On Cisco Catalyst 6000 with MSFC

What I thought was going to be a simple 5 minute procedure ended up taking 3 hours! My goal was very simple - add some security amongst a set of VLANs on a Cisco 6000 (CatOS 7-6-17) switch based on the following requirements:

-VLAN 30 cannot initiate connections to VLAN 50 and VLAN 60 but must get to the Internet
-VLAN 50, is the Admin VLAN and can get to everything
-VLAN 60 cannot initiate connections to VLAN 50 and VLAN 30 but must get to the Internet

Piece of cake, right? I quickly define the filters on the MSFC, deciding, as it should be to place the filters closest to the initiating traffic on VLANs 30 and 60 and have no filters on VLAN 50. The configuration turned out to be like this:

interface Vlan30
description Corp VLAN
ip address
ip access-group 130 in
interface Vlan500
description Admin VLAN
ip address

interface Vlan600
description Test VLAN
ip address
ip access-group 160 in

ip route

access-list 130 permit tcp any any established
access-list 130 deny ip
access-list 130 deny ip
access-list 130 permit ip any any

access-list 160 permit tcp any any established
access-list 160 deny ip any
access-list 160 deny ip any
access-list 160 permit ip any any

We begin testing this and yay, the traffic from VLAN 30 to 50 is blocked and....then... wait a minute now it's not blocked. I check the hit counter on ACL 130 and there were a couple denies from to but nothing more. What's going on here? We test from VLAN 30 to VLAN 50 and then from VLAN 60 to both VLAN 30 and VLAN 50 - the same result. The first packet observe the filter rules but thereafter access to each VLAN from any other VLAN is wide open. Grrrr!

At this point I know that my filters are good - there's no rocket science to setting these so I figured that it must be beyond just the filter that was causing this. The clue that led me to the solution was the result of a trace route from a source device ( to The icmp packet got there in just one hop! Now I expected the VLAN interface hop first before getting to the destination. This wreaked of a cached entry somewhere. But, how did it bypass the Layer 3 routing functionality or...did it? And then it hit me! My theories of Multilayer Switching (MLS) came flooding back and I remembered how the switch consults the router for the initial routing of the packet (Router on a Stick) and then caches the Layer 3 entry in the switch itself for subsequent routing of the paket (Layer3 switching). Of course!

Remember, I had not worked on a Cisco 6000 switch for years so I had to do some digging and found that MLS is enabled by default on this switch (with an MSFC of course). To disable it, merely add the 'no ip mls' command to each VLAN interface. So for VLAN30 it looked like this:

interface Vlan30
description Corp VLAN
ip address
ip access-group 130 in
no ip mls

I did this on all three layer 3 interfaces and the filters are now doing the work it was supposed to. You can clear the MLS cache by issuing clear mls on the switch itself.

Switch Details

WS-C6009 Software, Version NmpSW: 7.6(17)
System Bootstrap Version: 5.3(1)
System Boot Image File is 'bootflash:cat6000-supk8.7-6-17.bin'




ROM: System Bootstrap, Version 12.1(3r)E2, RELEASE SOFTWARE (fc1)

cisco Cat6k-MSFC2 (R7000) processor with 458752K/65536K bytes of memory.
Processor board ID SAD04450FV5

Thursday, March 26, 2009

These Cisco ASA VPN Filters Could Trip You Up!

This filter put a little twist into my powers of reasoning but I finally figured it out.

In our example, a Cisco ASA 5510 is serving as a VPN concentrator to which we have built a LAN to LAN IPSEC Tunnel from a customer peer ( Basically, a host on the customer end ( transfers files over the NDM protocol (tcp/1364) to our server ( and sometimes our server ( has to do the same to their server.

Summary of traffic over the tunnel: --> (tcp/1364) -->

Our aim is to build the VPN tunnel when traffic between the two servers is present but limit the traffic to just the NDM protocol.

Configuration Steps

1. First you want the VPN traffic to bypass the filter you have on the interface that the tunnel terminates because we don't want another filter for this traffic. In our case this is the 'untrust' interface. Use this command:

sysopt connection permit-vpn

2. Now define the encryption domain for the tunnel and the Phase 1 (ISAKMP) and Phase 2 (IPSEC) parameters. Whenever, the ASA 'sees' this traffic, it will start the process of bringing up the VPN tunnel.

access-list kis1-vpn-traffic line 1 extended permit ip host host

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto map untrust_map 10 match address kis1-vpn-traffic
crypto map untrust_map 10 set peer
crypto map untrust_map 10 set transform-set ESP-3DES-SHA
crypto map untrust_map 10 set security-association lifetime seconds 28800
crypto map untrust_map 10 set security-association lifetime kilobytes 4608000
crypto map untrust_map interface untrust

crypto isakmp identity address
crypto isakmp enable untrust
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400

3. With code 7.x on, you need to define tunnel group parameters first before the IPSEC tunnel will establish. This is where you define the pre-shared key and the filter that acts on the traffic within this tunnel

access-list kis1-vpn-filter line 1 extended permit tcp host eq 1364 host
access-list kis1-vpn-filter line 2 extended permit tcp host host eq 1364

group-policy kis1-group-policy internal
group-policy kis1-group-policy attributes
vpn-filter value kis1-vpn-filter

tunnel-group type ipsec-l2l
tunnel-group general-attributes
default-group-policy kis1-group-policy

tunnel-group ipsec-attributes
pre-shared-key *

4. Now let's take a closer look at the filter 'kis1-vpn-filter'. It has the familiar format where in this case, the source address is the far end address (customer host), while the destination address is the local host ( Line 2 is very logical - it is allowing their host ( to send traffic to our host ( on tcp/1364. Nothing wrong here.

The confusion arrives in Line 1. This line permits the local host to send traffic to their host on tcp/1364! So it appears that the filter does not act on the traffic leaving the local host to the remote host, but acts on the return traffic only. Without this you will not make a connection from your local host to the remote host. Seems easy now... :o)

How to Configure RSPAN on Catalyst 6500 IOS Switches

In our example we have two Cat switches (SWa and SWb). We have a packet sniffer connected to SWa interface F5/15 and want to monitor all the traffic going in and out of interface F3/1 of SWb.


1. Both switches must be trunked together.

2. Create an RSPAN VLAN. Preferably on the switch with VTP in server mode. Pick a VLAN ID that it is not being used currently. eg 999. If you do not have VTP, just create the VLAN on both switches.

Router(config)# vlan 999

3. Create an RSPAN source session on SWb. This is the traffic you want to capture. The session number can be any integer that's not being used for other sessions. All traffic (tx and rx by default) from F3/1 now gets copied onto VLAN 999

Router(config)# monitor session 2 source interface f3/1
Router(config)# monitor session 2 destination remote vlan 999

4. Create the RSPAN destination session on SWa.This session need not have the same session ID as the one on SWb.

Router(config)#monitor session 8 source remote vlan 999
Router(config)#monitor session 8 destination interface f5/15

5. And that's it! You can obviously expand to monitor source VLANs, multiple interfaces etc. The source session describe in step 3 works even if it is configured on the same switch as the destination session - but then it's technically not RSPAN!

6. Here's a network management tip. Have your packet sniffer connected permanently to an interface on the switch and set up that interface as the destination session. So whenever the need arises to monitor packets on any interface, vlan etc., all you've got to do is configure the source session. This way you will save loads of unnecessary time trying to figure out which interface your sniffer was plugged into! And, always remember to label the interfaces with the description command.