Now that I’ve shaken the cob webs off of the liver stress test I performed Sunday how about we get under the sheets with QoS? I’ve mentioned in a previous post that Class of Service (cos) is the layer 2 mechanism used for quality of service and that Differentiated Services Code Point (dscp) is the layer 3 mechanism for quality of service. The easiest way to grasp QoS for myself was to do a network trace.
Below is part of a header of a frame that has no QoS. You’ll see the Differentiated Services field is showing all zeros. That’s a pretty good indication that this frame has no priority.
Below is part of a header that has dscp markings for QoS. Here you can see the Differentiated Services field has some value. If you whip out your trusty calculator and set it to hex you’ll see that 2e in decimal is 46. That just happens to be the L3 Audio settings for QoS on our Avaya IP phones.
Below is part of a header that has dscp and cos values. Our Avaya IP phones have their L2 Audio values set to 6 (this is the default). In this frame the dscp value is 0×30 which converts to 48 in decimal.
If you are paying attention you probably noticed that the two frames with QoS markings have different dscp values even though they came from two identically configured Avaya IP phones. The reason is quite simple. The second header is from a frame that is the source and has just entered the switch. The bottom header is from a frame where my phone is the destination and the switch has already worked some of its magic. What magic is that? When QoS is enabled on a switch it has to have a way to convert layer 2 QoS to Layer 3 QoS. If you do a “show qos maps” on a Cisco switch you will see the CoS-DSCP Mapping Table. By default, Cisco maps Class of Service of 6 (Avaya L2 Audio default) to DSCP of 48. In the bottom header example the phone sent its traffic with a CoS value of 6 and the switch mapped it to a dscp value of 48.
Hopefully this makes QoS a little clearer for some. If nothing else it helped me by spelling it all out. On a completely unrelated note I started following this training guide for running a marathon tonight. I have no intention to run a marathon but I would like to increase my distances. I did the first 3 mile run tonight. Who knows, maybe I’ll do something insane and run a half marathon. Then again, maybe not!
Mr. Bump: “If I drank 15 beers tonight I’d be face down!”
Mrs. Bump: “Well maybe you should put your face down.”
After being elbow deep in QoS literature and network traces the last two days I’ve got a pretty good handle on the basics of it. What I don’t understand is if voice class of service (layer 2) is supposed to be 5 and the dscp (layer 3) value is supposed to be 46 why the hell does Cisco and Foundry make their cos 5 map to dscp 40 by default? Seems like mapping 5 to 46 would make better sense to me.
This afternoon we met with the Netscaler folks from Citrix. Or is it Xenscaler now? It was my first formal presentation of that product and it looks like a very robust appliance if you get the Platinum licensing.
Holy cow is life a lot less stressful tonight than it was last night. The Check Point issue I wrote about last night is resolved. Few things make you feel more like you are on an island than not being to modify your firewall policies or view the logs. The issue ended up being a registry key that was referencing a certificate path for a previous version of SmartCenter. Once I modified that path to the correct location I was able to revoke the current cert and create a new one. Upgrades are funny…a clean install is always preferable!
Now that that is behind me I can focus on the next thing on my list of “things I don’t know nearly enough about.” QoS. So far I’ve learned that DSCP is the layer three QoS mechanism and CoS is the layer 2 QoS mechanism.
Like a buddy of mine always says…I’m coachable!
On a personal note, thought I’d mention that our good buddy (and co-worker) El Sidekick lost his mother yesterday. You and your family are in our thoughts.
My brain is fried. The first two days of this week have been brutal. It started out wonderfully yesterday 
morning when the SSL certificate for our Check Point SmartCenter console expired over the weekend. I’ve spent a good chunk of both days dealing with that. The management server is an old Windows 2000 Server box that has been upgraded from NG to NG R55 to NGX R60. Now its a paper weight. TB threw out maybe it was a result of “Windows rot” and I think its a result of “Check Point Admin rot.” I’m not afraid to admit that my knowledge of exactly how Check Points interworkings go together is not that good. I’m learning though! School of hard knocks.
Speaking of something else I’m not that knowledgable about is QoS. But that too I am learning…from that same school too! The school of hard knocks. A local Cisco SE was out today to go over a few things regarding our infrastructure and I ran something by her regarding QoS. That was a mistake for my current mental capacity. She might as well have been speaking Portuguese to me. I had to jump off the Check Point train and jump on the QoS train. The University of Google is a much easier school than that hard knocks university! So the brain smokes while I sit in a chair yearning for a life less complicated. Feel sorry for me? Didn’t think so. I wouldn’t either.
Gonna find my baby, gonna hold her tight
gonna grab some afternoon delight.
My motto’s always been; when it’s right, it’s right.
How’s that for the worst intro to a post ever? I did manage to take this afternoon off and partake in a little socialization in the Power and Light district of KC. I hadn’t been down there yet so it was all new to me. I have to admit its a great step forward for Kansas City. It almost seems weird to have that kind of area now beings downtown has been a ghost town for so long.
Todays reward did not come about unearned. The crazy DHCP problem I’ve written about in the past showed up again yesterday. Fortunately some other odd behavior was reported along with it this time. In addition to the odd DHCP lease issue there were a lot disconnects reported. Yesterday afternoon TB noticed a couple of instances of this problem being reported for a department being in the incorrect VLAN. I thought nothing of it because it was only a couple and cables had been moved around so much in the troubleshooting of this problem that a port or two being in the wrong VLAN didn’t seem unreasonable. This morning another piece of the puzzle was added when we got actual visual confirmation of an endpoint getting an IP address for a subnet different than what the port was configured for. A little “debug dhcp detail” on the switch and the puzzle became a little clearer. The log and terminal monitor revealed this…
%CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet2/20 (210), with CSC06INDWC FastEthernet3/29 (147).
Looking at this it doesn’t take a rocket scientist to realize something was a miss. What was on port 2/20? What was on port 3/29? “Show cdp neighbor” put the final piece of the puzzle in place. Strangely this switch was a neighbor to itself…twice. It was connected to itself on port 2/20 and 3/29. Ahhh HA! A loop. This native VLAN confusion exposed another issue, a misconfigured VLAN interface. The VLAN interface had “ip-helper address” statements referrencing DHCP servers in a different location. The loop was sending DHCP requests to the incorrect scope at a different location which in turn was filling up the DHCP scopes. This explained the inability to obtain a DHCP address that would randomly appear.
As Tigger would say…this mystery is history!
It is really quite comical how things work out. Today we had a “mini” disaster recovery test for our mainframe. Like most things, every DR test comes with a couple of meetings prior to the actual work being done. The part I’m involved in is really pretty simple and only involves ensuring a VPN tunnel to the recovery site comes up. The simple part is the config for this VPN was created back in 2005 and our end has not changed since. Every test comes with the same sense of importance stressed on this VPN creating successfully. I get asked several times if somebody will be available at the time the test starts to ensure the VPN comes up. I give the same bullshit answer every time…”Of course, if I’m not here El Sidekick is here at 6:00 AM every day so he will be here.” I give the same answer because I know that if the test is scheduled to start at 8:00 AM they won’t be to a point where they need the VPN until 2 hours later at the earliest.
Today was no different. The test was scheduled to start at 8:00 AM. Shortly after 2:00 PM they were at a point where they needed the VPN. The VPN was up at 8:15 and sat idle for 6 hours. The tardiness of the VPN creation wasn’t even an issue on our end. For the third test in a row the DR site clowns didn’t put a default gateway on their config. Silly clowns.
This has been a crazy busy week. The stars must have been aligned this week to cause me to be this busy. I think I’m going to reward myself by taking Friday afternoon off. If I can’t blow it out on a Friday when the hell can I blow it out?
Ever heard the saying “busier than a one legged man in a butt kicking contest?” My buddies who I normally fire emails back and forth with throughout the day are wondering if I am still alive.
Today we began the “discovery phase” of what is necessary to do on our Citrix farm in order to put our Citrix servers on our LANenforcer 1048 secure switches that were delivered about 10 days ago. This comes at kind of a crossroads for our Citrix environment as we are in need of a server refresh at the very minimum and probably an entire farm rebuild if we really wanted to do things right. We also are going to pilot an application streaming implementation at the recommendation of Server Centric after a mini health check on our Citrix farm. I think we have some very interesting things going on in the near future with our Citrix direction and the introduction of our LANenforcers to that environment.
On another Citrix note it looks like I will be attending Citrix Synergy May 20-23 in Houston. I attended iForum in 2005 in Las Vegas and almost got divorced during that trip. I’m playing this conference safe and leaving the wife at home! What I’m really looking forward to is the virtualization addition of the conference after the acquisition of XenSource.
The call for routing experts is no longer needed. Friday night I posed the brain buster from the previous post in a Cisco forum looking for answers. It took all of an hour to have an answer…a rather obvious one at that. The reassuring part is that I was on the right track Thursday night when I made the post. I knew what the problem was Thursday night but made a common troubleshooting mistake. Instead of analyzing the problem for what it was I was focusing on why it worked on one router and not on the other.
The issue was an ARP request for an IP address that no interface owned. The obvious solution? Change the NAT statement to NAT to the IP configured on Fa 0/1. Duh. A proxy ARP could have been configured as well. I knew this. I even attempted to configure a sub-interface on Fa 0/1 Thursday night. The router promptly slapped me in the face and reminded me that sub-interfaces can only be configured on trunks. A little thinking about the problem and I would have solved this Thursday night. Stupid is as stupid does.
The brain buster is a brain buster no more. Troubleshooting 101 and I failed miserably on this test!
I’ve got a real brain buster on my hands. Its a real brain buster for me anyway! We have two routers: a Cisco 1721 and a Cisco 2620. Right now the 1721 is doing policy based routing to route traffic from a certain IP address (1.1.1.2) out a different internet link. If I put the exact same config on the 2620 router, traffic to the second link does not flow. Network traces show the routing and NAT working but the next hop (2.2.2.1) does not return the traffic. I can see the ARP request from 2.2.2.1 but the 2620 does not answer. Below is the relevant config of the 2620.
interface FastEthernet0/0
ip address 1.1.1.1 255.255.255.240
ip nat inside
ip policy route-map Policy1
interface FastEthernet0/1
ip address 2.2.2.2 255.255.255.248
ip nat outside
ip nat inside source static 1.1.1.2 2.2.2.3
ip route 2.2.2.0 255.255.255.248 2.2.2.1
access-list 2 permit 1.1.1.2
route-map Policy1 permit 10
match ip address 2
set ip next-hop 2.2.2.1
About the only thing different other than the router model is that the 1721 only has one Fast Ethernet interface on it so it has a 4-port Fast Ethernet Switch WAN Interface Card installed in it. That means that the second interface is actually a switch interface with the IP address (2.2.2.2) being a VLAN interface on the 4-port switch. Make sense?
On another note…the problem I wrote about a couple of days ago about the constant FTP connections from a third party has been resolved. Turned out that they had a hung ftp process on their that was constantly making connection attempts. As soon as they stopped it they were able to make successful connections. Its always something.