^ Top

NANOG Meeting Presentation Abstract

Interesting Peering Activities at the Exchange Points
Meeting: NANOG14
Date / Time: 1998-11-09 11:30am - 12:00pm
Room:  
Presenters: Speakers:
Naiming Shen, Cisco Systems.
Abstract: During the summer of 1997, when I was working on Internet peering issues at iMCI, I had a chance to help track down a couple of unusual peering activities. These involved rewriting eBGP next hops to some NAP routers; passing third party next hops; pointing default; and registering incorrect DNS names for NAP routers. Some of these activities were due to a misconfiguration, such as running IGP protocols over NAP FDDIs or turning on native IP multicasting on the NAPs.

Case #1: Rewriting eBGP Next hops At this time iMCI had two routers at MAE-East, called cpe2 and cpe3. I was informed by the NOC that the cpe2 FDDI inbound was very congested. I turned on the Netflow feature on iMCI\'s routers and found that 15% of the incoming traffic from cpe2 was unaccounted for. This meant that the traffic was coming from someone iMCI did not peer with at MAE-East. Further analysis showed that almost all the traffic was coming from a single subnet.



The BGP routing table showed that that subnet was 3 AS hops away from iMCI. iMCI had established private peering with ISP-1, ISP-1 peered with ISP-2, and ISP-2 peered with ISP-3, which owned the subnet. iMCI did not peer with any of these ISPs at the NAPs, so the only way ISP-3 could point next hop to iMCI was to rewrite the next hop by matching iMCI\'s AS number in its eBGP routes.



I placed an ACL packet filter on cpe2 to block traffic from ISP-3. Luckily. ISP-3 only had a single block of addresses, which made it possible to do packet-level filtering. ISP-3 then changed its routing to ISP-2 and ISP-1. After I removed the packet filtering, however, ISP-3 pointed its traffic back to iMCI again.



I then decided to create something even more interesting. I designed a filter to let ISP-3 pass ICMP packets, traceroute packets, and DNS packets, but block all other IP packets, just to add some complexity to their troubleshooting. The filter was there for four days; apparently they could not pinpoint the problem, and finally switched traffic back to normal.



Case #2: Passing Third-Party Next hop This case was not flagged by the Netflow analysis, because the reverse path lookup did not fail. iMCI peered with ISP-4 at MAE-East, but not with ISP-5. ISP-4 not only passed our next hop to ISP-5, but also passed ISP-5\'s next hop directly to us. In this way, we exchanged traffic with ISP-5 directly. We could, of course, manually overwrite all the next hops to ISP-4\'s address, but we still could not stop ISP-4 from passing our next hop to ISP-5. After exchanging some email with ISP-4, they agreed to fix this. Some might believe that it was more efficient to pass traffic between third parties over multi-access media. However, I believed that peering was not just an engineering issue, but also a business issue.



Case #3: Pointing Default Some routers at the exchange points simply pointed default to others. One strange example was a router at MAE-East that pointed default to UUnet; the router name had a reverse DNS lookup of xxx.internetmci.net. UUnet therefore asked me if this was one of our routers. I knew that we registered mci.net and internetmci.com, but was not sure about internetmci.net. A couple of days later, the IXP router pointed default to us instead of to UUnet. Since they claimed to be MCI, we did an SNMP query to the router, and it returned:



ip.ipRouteTable.ipRouteEntry.ipRouteNext

hop.0.0.0.0=IpAddress:192.41.177.180 +



The ip address was our cpe3\'s FDDI address at MAE-East. I also obtained the AS number of the router, and was able to find out who the router belonged to. After several email exchanges, the owner changed their default route so it did not point to us any more, but to someone else ;-) Somehow, they believed a router had to have a default route.


Other Activities



Other issues were debatable. I found some ISPs running IGPs over the FDDI at the NAPs. When I sent the ISPs e-mail, they told me that this was due to a misconfiguration. One could argue this was a way to create redundancy at the NAPs. But usually an ISP\'s routers were at the same location, or on the same rack at the NAPs, so redundancy could be achieved by using a private LAN instead of bothering other routers.
One or two routers/workstations on MAE-East were using native IP multicast. I calculated that about 2Mbps of this traffic was going over the NAP FDDI, but didn\'t know if any router on the LAN was receiving the traffic. I talked to the senders, who told me that they planned to shut down multicast on their box.



I spent a lot of time dealing with route consistency over different peering points. On the Internet today, we use shortest exit routing. If the routes we learned from our peers were not consistent across all the peering points, then we might carry some traffic unnecessarily across our backbone. Sometime we had to avoid some peering points because of severe congestion; this was usually done through mutual agreement.



To deal with those unusual activities, we need to detect them, communicate with the right people, and, sometimes, find a way to stop them. To detect problems, I was able to use:

  • Netflow statistics for reverse route lookup

  • Traceroute with \"-g\" switch

  • If LSR is disabled on the other end, and you are trying to investigate how that router route certain prefixes, you can temporarily create a static route pointing to the router in question, and then trace to that address. If that router routes that trace packet back to you, then you will see the trace going back and forth between your router and the other router.

  • MAC address accounting.




Methods you can use to stop unwanted traffic include:

  • Packet level filtering if the router CPU can handle it

  • MAC address filtering/rate-limit, sometimes combined with WRED

  • Filter out routes from the network if necessary




Preventive practices include:

  • NAP GIGAswitch L2 filtering
    Use next-hop-self and always overwrite the next hop to your peer\'s address.

  • If you don\'t have a customer on the NAP routers, remove the non-customer routes from the NAP routers. This ensures that you only allow peers\' traffic to go to your customer destinations.


  • Use the loopback address to do iBGP peering, so you don\'t have to carry NAP LAN address blocks over the network.

  • Use ATM PVCs

Files: pptShen Presentation(PPT)
Sponsors: None.

Back to NANOG14 agenda.

NANOG14 Abstracts

  • Technology for Backbone Web Caching
    Moderators:
    Peter DanzigNetwork Appliance; .
    Panelists:
    James AvianiCisco Systems; .
    Shirish SathayeAlteon; .
    Bill MaggsMCI; .
    Speakers:
    Ed KernDIGEX; .
  • Technology for Backbone Web Caching
    Moderators:
    Peter DanzigNetwork Appliance; .
    Panelists:
    James AvianiCisco Systems; .
    Shirish SathayeAlteon; .
    Bill MaggsMCI; .
    Speakers:
    Ed KernDIGEX; .
  • Technology for Backbone Web Caching
    Moderators:
    Peter DanzigNetwork Appliance; .
    Panelists:
    James AvianiCisco Systems; .
    Shirish SathayeAlteon; .
    Bill MaggsMCI; .
    Speakers:
    Ed KernDIGEX; .
  • Technology for Backbone Web Caching
    Moderators:
    Peter DanzigNetwork Appliance; .
    Panelists:
    James AvianiCisco Systems; .
    Shirish SathayeAlteon; .
    Bill MaggsMCI; .
    Speakers:
    Ed KernDIGEX; .
  • Technology for Backbone Web Caching
    Moderators:
    Peter DanzigNetwork Appliance; .
    Panelists:
    James AvianiCisco Systems; .
    Shirish SathayeAlteon; .
    Bill MaggsMCI; .
    Speakers:
    Ed KernDIGEX; .

 
^ Back to Top