on Mar 25th, 2008Chasing the Dragon
Today started well enough. We had two people at a Microsoft Server 2008/SQL 2008/Some Developer Crap 2008 launch event and it was setting up for a nice quiet day with a short staff. Then about 9 o’clock the wheels fell off. I’ve posted a couple of times, here and here, about this crazy DHCP issue that popped up out of nowhere. Based on what we were seeing on our LANenforcer 2024 we were confident that the DHCP requests were never leaving the switch as the traffic was not being detected by our NAC appliance (its a bump in the wire incase you have not been paying attention). Over the course of three or four days we moved roughly 10 users each day off this single problematic switch. The problem went away just as fast as it appeared. Back to this morning. Last night we moved 28 of the users that had issues nearly three months ago back to the switch they were on when it happened. This morning about 9:00 the phone rang off the hook with users that could not get DHCP addresses again. These were different users this time and not the ones that were moved last night. So today was spent chasing down this problem. Right now we have no answers. I ran this problem by a local Cisco SE and she thinks it might be a spanning-tree hold-down issue. According to her a DHCP issue similar to this a very common “side effect” of having spanning-tree enabled. I guess we will find out manana when we expect this problem to return and we are able to enable “spanning-tree portfast” on one of these ports. I guess I’ll have to quote Hoff from his post today on this one…
“No offense to my brethren in the trenches, but this is simply a case of experience and expertise. Server admins are not experts in network or security architectures and operations, just as the latter cannot hope to be experts in the former’s domain.”
Guilty. This issue has proven that if nothing else. But I have to do both!
Over at StillSecure, After All These Years, Alan wrote a post a few days ago about blogging for the sake of blogging. This reminded me of a great movie, Pump Up the Volume. Only instead of pirate radio we get to express our thoughts and opinions in a blog on the silly internet. Its on HBO2 at 4:00 AM tomorrow morning. DVR is set to record.
Blog Hard!!
Did you ever figure out what was going on?
Reminds me of the problem we’re seeing with IPv6 -enabled Vista machines coming onto a network with a rogue v6 DHCP server. I think the general consensus has been to block IPv6 on the switch(es) as a just-in-case.
-jj
Never did find out. Was advised to open a TAC case with Cisco. With 25 less users on this switch the problem does not occur. Very strange.