North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Re: Weird network problem.
Hey Bob The problem we were seeing was in fact different than the one you seem to have. It was specific to IE and was corrected by a later MS patch. If it is any help, I can give you an account on the network to work with in your diagnosis. Send me an email off list or call. I notice in your documentation that you mention a "conversion to the 10.x.x.x" network. Are you NAT'ing? Herman Harless Director Advanced Data Services NTELOS, Inc. On Fri, 2004-03-12 at 10:06, Bob Harden wrote: > > Has anybody on the list ran into any problems like this? We don't have > a firewall and have tried with and without our ACL's but get the same > results. Below is our internal white paper on the situation. If > anybody has any ideas they would be much appreciated. > > Thanks, > > Bob Harden > > > > > Network problem information. > > Symptoms: At random times dialup, dedicated, & internal network users > are unable to > pass TCP traffic to off network sites. ICMP and UDP appears > to be > uneffected by the outage which lasts anywhere from 2 to 5 > minutes. > > The problem appears to be wide spread with similar reports > from WVNET > and other ISPS. nTelos is experiencing a similar problem but > we have > yet to confirm it is the same. > > Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, & 2003 Server. > > > Uneffected Platforms: Unix & MacOS > > > History: During the week of 2/9/04 the call center started recieving > reports of > users being unable to connect to sites off the network. Sites > hosted on the internal network are uneffected by the outage. > > Initally it was thought to be a Internet Explorer problem > possably caused > by the KB832894 / IE SP1 or other updates but after further > investigation > it was found that Mozilla users were encountering the same > problem. > > After several days of testing it was determined that during the > outage any > TCP session started on any port would fail. TCP sessions > started before > the outage continue to work and show no ill effects from the > outage. > > After logging connection attempts at various intervals on many > machines > there appears to be no sort of pattern in the outages. Most > machines > encounter the problem, some more than others and a few do not > encounter > it at all. The duration and frequency of the outage is very > fluid. > > During an outage, we can verify that the packet does seem to > leave and reenter > the network: > > Mar 5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h: > %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) -> > 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17084: SLOT > 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> > 69.43.14.174(3376), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17085: > SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp > 69.43.14.174(3378) -> 216.41.224.3(80), 1 packet Mar 5 22:28:09 > pittpa-chaswv-ds3 17086: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 > permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 1 packet Mar 5 > 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: > list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 7 packets > Mar 5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h: > %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> > 69.43.14.174(3376), 17 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17092: > SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp > 69.43.14.174(3378) -> 216.41.224.3(80), 7 packets Mar 5 22:33:58 > pittpa-chaswv-ds3 17093: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 > permitted tcp 69.43.14.174(3376) -> 216.41.224.3(80), 18 packets > > Mar 5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h: > %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) -> > 216.41.224.3(80), 1 packet Mar 5 00:58:30 pittpa-clarwv-ds3 16063: SLOT > 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> > 69.43.23.23(3183), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16067: > SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp > 69.43.23.23(3217) -> 216.41.224.3(80), 1 packet Mar 5 01:03:28 > pittpa-clarwv-ds3 16068: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 > permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 1 packet Mar 5 > 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: > list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 1 packet > Mar 5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h: > %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> > 69.43.23.23(3228), 1 packet Mar 5 01:03:39 pittpa-clarwv-ds3 16072: > SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp > 69.43.23.23(3239) -> 216.41.224.3(80), 1 packet Mar 5 01:03:47 > pittpa-clarwv-ds3 16073: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 > permitted tcp 69.43.23.23(3183) -> 216.41.224.3(80), 74 packets Mar 5 > 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: > list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3183), 72 packets > Mar 5 01:08:46 pittpa-clarwv-ds3 16078: SLOT 2:5d22h: > %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3218) -> > 216.41.224.3(80), 4 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16079: > SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp > 69.43.23.23(3217) -> 216.41.224.3(80), 3 packets Mar 5 01:08:46 > pittpa-clarwv-ds3 16080: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 > permitted tcp 69.43.23.23(3221) -> 216.41.224.3(80), 19 packets Mar 5 > 01:08:46 pittpa-clarwv-ds3 16081: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: > list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 5 packets > Mar 5 01:08:46 pittpa-clarwv-ds3 16082: SLOT 2:5d22h: > %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3229) -> > 216.41.224.3(80), 6 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16083: > SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp > 69.43.23.23(3236) -> 216.41.224.3(80), 9 packets Mar 5 01:08:47 > pittpa-clarwv-ds3 16084: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 > permitted tcp 69.43.23.23(3233) -> 216.41.224.3(80), 12 packets Mar 5 > 01:08:47 pittpa-clarwv-ds3 16085: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: > list 113 permitted tcp 69.43.23.23(3239) -> 216.41.224.3(80), 21 packets > Mar 5 01:09:12 pittpa-clarwv-ds3 16087: SLOT 1:5d22h: > %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> > 69.43.23.23(3239), 19 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16088: > SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp > 216.41.224.3(80) -> 69.43.23.23(3228), 4 packets Mar 5 01:09:12 > pittpa-clarwv-ds3 16089: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 > permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 2 packets Mar 5 > 01:09:12 pittpa-clarwv-ds3 16091: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: > list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3218), 3 packets > Mar 5 01:09:13 pittpa-clarwv-ds3 16092: SLOT 1:5d22h: > %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> > 69.43.23.23(3221), 17 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16093: > SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp > 216.41.224.3(80) -> 69.43.23.23(3229), 5 packets Mar 5 01:09:13 > pittpa-clarwv-ds3 16094: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 > permitted tcp 216.41.224.3(80) -> 69.43.23.23(3236), 7 packets Mar 5 > 01:09:13 pittpa-clarwv-ds3 16096: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: > list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3233), 9 packets > > > Network analysis showed significant amounts of spoofed > multicast traffic and > odd arp traffic. > 10:17:16.416222 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 278 > 10:17:16.421886 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 334 > 10:17:16.423873 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 262 > 10:17:16.426948 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 254 > 10:17:16.432095 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 298 > 10:17:16.435921 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 274 > 10:17:16.439959 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 328 > 10:17:16.445317 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 326 > 10:17:16.449688 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 330 > 10:17:16.463537 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 322 > > Steps were taken to elminiate the spoofed traffic on the > routers and access servers in the form of ACLs and filter > lists. > Neither have eliminiated the problem... but to what extent they > might > have helped has yet to be determined. > > The problem is still occuring, for some users the duration of > the outage > seems to have shortened other users notice no difference. It > is not yet > known if the filtering on the routers and access servers or the > conversion > to the 10.x.x.x network has made any difference. We should > have a better idea > in the upcoming days. > > What Doesn't help: Removing windows updates. > Turning off XP firewall. > Searching for malware. (SpyBot-SD, Adaware) > Virus scanning (Various softwares) > Specifying dns servers. > Reinstalling windows. >
|