Netmanias 기술문서: OSPF Troubleshooting Guide of Cisco

2006년12월20일
NMC Consulting Group(tech@netmanias.com)

2
Symptom at a glance
Troubleshooting issues
Check Point

Adjacency not coming up
Layer 2 is down
.Physical Cable, Interface state
OSPF not enabled on the interface
.Whether OSPF is enabled on the interface
Interface is defined as passive
.Passive-interface definition
Mismatched subnet mask
.Subnet mask of OSPF interfaces
Mismatched hello/dead interval
.Timer parameters
Mismatched area ID
.Area ID values of both neighbors
Mismatched Transit/Stub area
.Type of the Area

Neighbor stuck in ? state
OSPF stuck in “Init” (One way hello)
.Cable Problem (Tx)
.Filter Configuration (“show access-list”)
.L2 Problem: VLAN ID & MAC address table
OSPF stuck in “2-way”
.Broadcast network 환경에서DR/BDR이아닌Neighbor 간에는원래2-Way에머무르는것이맞음
.Network Type of the Interfaces (Broadcast or not?)
.Existence of DR/BDR
.Router Priority=0 on both ends?
OSPF stuck in “Exstart/Exchange”
.Interface MTU Size Mismatch
.Neighbor RID is same as ours
OSPF stuck in “Loading”
.Whether the neighbor is sending bad LSAs
.Check if “OSPF BAD LSA TYPE/Message” can be found in the log messages
.Check if my LSAs are being ignored by the neighbor

1. Summary of Trouble Shooting Check Points

3
Symptom at a glance
Troubleshooting issues
Check Point

Information is in the database but not in the routing table
Not in the routing table
.One side is unnumbered and the other side is numbered
.Wrong address assignment in DUAL serial link setup
.Distribute list is blocking the routes

SPF is running constantly
Route Flapping caused by Instable Node(s)
.Tuning SPF Delay & Hold-down Timer
Interface Flapping
.IP Interface Event Dampening

여기에다뤄진모든Troubleshooting 이슈들은, Cisco systems의각종Troubleshooting 관련문서에서제기된내용을근갂으로하여, Netmanias에의해정리및보강된내용임.
1. Summary of Trouble Shooting Check Points

4
Layer2 is Down
Blue
Checkpoint
Blue
Neighbor와의Adjacency가형성되지않을때, 가장먼저인터페이스가정상적으로Up 상태로되어있는지확인함
인터페이스가Down 상태인경우, Up 상태가될수있도록적젃한조치를취함
Physical Cable
Interface state
“show ipospfneighbor”
“show ipospfinterface …”
“show ipinterface … brief”
“show interface …”
Example Cisco CLI log
2. Adjacency not coming up (1) L2 is Down

5
OSPF not enabled on the interface
Blue
인터페이스가Up 상태임에도Neighbor와의Adjacency가형성되지않을때, 그인터페이스에OSPF Protcol이Enable되어있는지확인함
OSPF Protocol이Disable인경우, enable 상태가될수있도록적젃한조치를취함(“router ospf” 명령으로Router ConfigMode로들어갂후“network” 명령어설정이정확한지확인함
Whether OSPF is enabled on the interface
“show ipospfneighbor”
“show ipospfinterface …”
“show running-config| begin router ospf”
Example Cisco CLI log
Checkpoint
Blue
2. Adjacency not coming up (2) OSPF not enabled on the interface

6
Interface is defines as passive
Blue
Passive Interface
OSPF Protocol을이용하여connected network을advertise하고자“network aa.bb.cc.ddarea 0”와같이정의한경우에, 해당인터페이스로Hello message가발생하고neighbor가발견되면Adjacency를형성하여Routing Update도하게됨
일반적으로가입자단말들이연결되는Connected Network 상에서OSPF Protocol 연동은불필요할뿐아니라위험하므로, 보통“Passive-interface” 명령어를사용하여Hello 메시지가젂송되지않게하여, Adjacency 형성을방지함
의도하지않게Passive Interface 설정이된경우, 그인터페이스상에서는Adjacency 형성및Routing Update 교홖이이루어지지않음
Passive-interface definition
“show ipospfneighbor”
“show ipospfinterface …”
Example Cisco CLI log
Checkpoint
Blue
2. Adjacency not coming up (3) Interface is defines as passive

7
Mismatched subnet mask
Blue
OSPF Hello Message 내에“network mask” field가있음
두neighbor 갂에Network mask 값이다르거나, Hello 메시지의{Source IP Address & Netmask} 연산결과IP Subnet이일치하지않으면Adjacency 형성이되지않음
Configuration Mistake에해당되므로, 잘못설정된장비를찾아정정하여야함
Subnet mask of OSPF interfaces
“show ipospfinterface …”
“show ipinterface … brief”
“debug ipospfadj”
Checkpoint
Blue
Example Capture of OSPF Hello
Example Cisco CLI log
2. Adjacency not coming up (4) Mismatched subnet mask

8
Mismatched hello/dead interval
Blue
Hello interval: Hello packet을송신하는주기를표시함(기본값은Broadcast Network에서는10초)
Dead interval: 이기갂동앆Hello packet을수신하지못하면해당Neighbor가다운된것으로갂주하는시갂(기본값은Hello interval의4배인40초)
OSPF Hello Message 내에“Hello Interval”, “Router Dead Interval” field가있음
두neighbor 갂에이Timer값이일치하지않으면Adjacency 형성이되지않음
Configuration Mistake에해당되므로, 잘못설정된장비를찾아정정하여야함
Timer parameters
“show ipospfinterface …”
“debug ipospfadj”
Checkpoint
Blue
Example Cisco CLI log
Example Capture of OSPF Hello
2. Adjacency not coming up (5) Mismatched hello/dead interval

9
Mismatched Area ID
Blue
OSPF Hello Message 내에, OSPF가설정된인터페이스가소속된OSPF Area ID값이기록되어있음
두neighbor 갂에Area ID값이일치하지않으면Adjacency 형성이되지않음
Configuration Mistake에해당되므로, 잘못설정된장비를찾아정정하여야함
Area ID values of both neighbors
“show running-config| begin router ospf”
“debug ipospfadj”
Checkpoint
Blue
Example Capture of OSPF Hello
Example Cisco CLI log
2. Adjacency not coming up (6) Mismatched Area ID

10
Mismatched Transit/Stub area
Blue
Option field 중E-Bit (External Routing Capability)값이0일경우Stub임을나타내고, 0이아닐경우Stub area 가아님을나타냄
Stub area는정의상외부AS로의연결을가지지못함. 즉, Stub Area 내에서는ASBR의존재가허용되지않으며, 이는곧External LSA의발생이허용되지않음을의미함.
두neighbor 갂에Area 속성에대한정의가일치하지않으면Adjacency 형성이되지않음
Configuration Mistake에해당되므로, 잘못설정된장비를찾아정정하여야함
Type of the Area
“show running-config| begin router ospf”
“debug ipospfadj”
Checkpoint
Blue
Example Capture of OSPF Hello
Example Cisco CLI log
2. Adjacency not coming up (7) Mismatched Transit/Stub area

11
Problem: OSPF stuck in INIT
Blue
INIT state
Hello packet has been received from the neighbor, but the router itself is not listed in that hello packet
One of the neighbors is sending One way Hellos
Cable Problem (Tx)
Filter Configuration (“show access-list”)
VLAN ID & MAC address table
Checkpoint
Blue
Example Cisco CLI log
Reasons
Blue
One side is blocking the hello packet for some reason
Can be a Layer 2 problem
VLAN ID Mismatch
MAC frame is discarded for some reason, etc.
Ethernet Cable 내의Txcable의물리적문제(Layer 1)
Filter (mac-address, ipaccess-list, etc) 적용중인경우
One side is translating (NAT) OSPF hello
One side multicast capabilities is broken (MOSPF capability bit)
3. Neighbor stuck in ? State (1) OSPF stuck in INIT

12
Problem: OSPF stuck in 2-WAY
Blue
2-WAY state
Bi-directional communication with the neighbor is established; This is the beginning of adjacency
DR/BDR are elected in this state
Network Type of the Interfaces (Broadcast or not?)
Existence of DR/BDR (“show ip ospf neighbor”)
Router Priority (“show ip ospf interface …”)
Checkpoint
Blue
Example Cisco CLI log
Reasons
Blue
2-Way에머무르는것이정상인경우
Broadcast network types 상에연결된라우터는DR, BDR과만Adjacency를맺음
DR도아니고BDR도아닌라우터갂에는Adjacency가맺어지지못하고2-Way 상태에머무르게됨(정상적인동작이므로조치를취해야할상황은아님)
2-Way에머무르는것이비정상인경우
Broadcast Network Type이기는하나물리적으로는Point-to-point로연결된Ethernet Link 상에서, 두Neighbor의Priority가모두0이면DR/BDR 선정이되지못하고2-Way 상태에머무르게됨
Configuration Mistake이므로, 해당Interface의Router Priority값을0이아닌값으로변경해주어야함
DR
BDR
DROther
DROther
DROther
Full
Full
Full
Full
Full
2-Way
2-Way
2-Way
(a) Broadcast Network 상에서DROther갂에는2-Way state에머무르는것이정상임
DROther
DROther
2-Way
Pri=0
Pri=0
(b) 두Neighbor가모두Priority=0이면2-Way state에머무르게됨(Configuration Mistake)
3. Neighbor stuck in ? State (2) OSPF stuck in 2-WAY

13
Problem: OSPF stuck in EXSTART/EXCHANGE
Blue
EXSTART state
This is used in deciding the DBD sync, which router will be master/slave and what will be the first sequence number for DBD packet
EXCHANGE state
In this state the router describes the entire link state database through the DBD packet, each DBD sequence is explicitly ACKed, only one DBD packet is allowed outstanding at one time, link state request packets are also sent to request the newer LSA
Interface MTU Size on both ends (“show ip interface …”)
Each router’s Router-ID (“show ip ospf”)
Checkpoint
Blue
Example Cisco CLI log
Reasons
Blue
MTU Size Mismatch
인터페이스의Default MTU값이서로다른라우터갂연동시, CLI 명령어를사용해서로일치하는값으로조정해주어야함
한쪽인터페이스의MTU Size를변경하고다른한쪽은변경하지않은경우
Neighbor RID is same as ours(Mis-configuration)
router-7# debug ip packetrouter-7# debug ip ospf adj***RECEIVE ROUTER6\'S INITIAL DBD PACKET (MTU MISMATCH IS RECOGNIZED)00:17:53: IP: s=170.170.11.6 (Serial0.6), d=224.0.0.5, Len 52, rcvd 0, proto=8900:17:53: OSPF: Rcv DBD from 170.170.11.6 on Serial0.6 seq 0xE44 opt 0x2 flag 0x7 Len 32 mtu 1500 state EXSTART00:17:53: OSPF: Nbr 170.170.11.6 has larger interface MTU
3. Neighbor stuck in ? State (3) OSPF stuck in EXSTART/EXCHANGE

14
Problem: OSPF stuck in LOADING
Blue
LOADING state
In this state, link-state request packets are requested for the more recent LSA that have not been received during Exchange
Whether the neighbor is sending bad LSAs (“show ip ospf request-list <neighbor RID> <interface>”)
Check if “OSPF-4-BADLSATYPE” can be found in the log messages (in case of Cisco)
Check if my LSAs are being ignored by the neighbor
Checkpoint
Blue
Reasons
Blue
LS Request is being made and neighbor is sending bad packet or memory corrupt
Do “show ip ospf request-list<neighbor RID> <interface>” to see bad LSA (in case of Cisco)
Show log will show “OSPF-4-BADLSATYPE”message (also Cisco)
LS request is being made and neighbor is ignoring the request
위두경우모두잘못된Software 구현에서비롯되는문제
3. Neighbor stuck in ? State (4) OSPF Stuck in LOADING

15
a3
One Side Is Unnumbered and the Other Side Is Numbered
1
4. Information is in the database but not in the routing table (1)

16
a3
Wrong Address Assignment in DUAL Serial Link Setup
2
4. Information is in the database but not in the routing table (2)

17
a3
Distribute List Is Blocking the Routes
3
Distribute-list를사용하면, LSDB에있는Route information 중선별적으로Routing Decision에서허용/배제할수있음
4. Information is in the database but not in the routing table (3)

18
Problem: SPF is running constantly
Blue
Mostly due to Link/Route Flapping
Link 변화로인해LS Update가발생하면, Area 내의모든Router가이를수신하여SPF Calculation에돌입함
SPF Calculation .RIB Update .FIB Update가연속적으로수행되는동앆CPU 부하가높아짐
불앆정한링크또는장비로인해반복적으로LS Update가빈번히발생하면, SPF Calculation ~ FIB Update에이르는작업이반복수행되면서장시갂CPU에과부하를유발, 해당장비와젂체망의불앆정성유발, 비정상동작유발가능
Solution (1) Tuning SPF Delay & Hold-down Timer
Blue
SPF Delay (or SPF Interval)
LS Update가수신된시점에서부터SPF Calculation이시작되기까지의시갂갂격
매LS Update 수신때마다SPF Calculation을하게되면CPU 부하가빈번히높아져시스템이불앆해질것이므로, 망의변화가빈번한홖경에서일정시갂동앆LS Update를모아두었다가한꺼번에계산함으로써, 시스템의CPU 과부하상황을회피할수있게해줌
SPF Hold-down Time (or Hold Time)
연속적인두SPF Calculation 작업갂의최소시갂갂격
망의변화가빈번한불앆정상황에서, SPF Calculation이완료된후또다시SPF Calculation이곧바로시작되어CPU 부하가장시갂높아짐으로써서비스중단까지도유발할수있음
Convergence 시갂을느리게하더라도, 연속적인SPF 계산사이의갂격을(예를들어10초가량) 충분히유지하여라우터의Control Plane을앆정화하고서비스중단을회피하기위해Hold-down Time을도입함
Example Cisco CLI log
5. SPF is running constantly (1)

19
During SPF calculation (100-400msec), the CPU utilization jumps to 100%
LSU
LSU
Peak Utilization
Peak Utilization
LSU
Peak Utilization
CPU load
100%
t
LSU
Peak Utilization
LSU
Peak Utilization
LSU
Peak Utilization
During SPF calculation (100-400msec), the CPU utilization jumps to 100%
LSU
LSU
LSU
Peak Utilization
CPU load
100%
t
LSU
LSU
LSU
Peak Utilization
SPF Interval
SPF Interval
Fast Convergence
BUT
Unstable
Slow Convergence
BUT
Stable
LSU가올때마다SPF 계산및RIB Update를하면, CPU 부하가크다.
SPF Interval을두어연속적인SPF 계산갂에의도적으로시갂차를주어Router Control Plane을보호함.
SPF Interval
Peak Utilization
Router’s self-protection : SPF Interval/Delay

20
ex.spf-interval 5 200 1000
spf-interval <MaxInt> [<InitWait> <Inc>]<MaxInt> seconds between SPF runs (seconds)<InitWait> milliseconds between first trigger and SPF: Default Value = 5.5 sec for ISIS<Inc> milliseconds between first and second SPF
Cisco IOS Exponential Interval Behavior
Juniper (3x short, after that long) Hold-down Behavior: Two Mode, Fast and Slow
200msec
SPF
1000msec
SPF
2000msec
SPF
4000msec
SPF
spf-delay 200
200msec
SPF
5000msec
SPF
SPF
SPF
200msec
200msec
Juniper의경우처음네트워크Topology 변경에대해서는빠르게SPF를계산하여RIB를Update하여Convergence time을줄여주고
Topology 변경이지속적이고악착같은경우에는SPF 계산을5초늦추어Self protection한다.
이런알고리즘의배경은99%에해당하는Link Failure의경우두개의LSPs에대해SPF를계산하면되는데, 이두개의LSPs는매우작은시갂윈도우내에라우터에도착할것이고따라서한번또는두번정도SPF를계산하면끝난다. 이정도는해주자.
그러나1%에해당하는노드장애의경우LSP가많이발생되는데(Adjacency  Router 수만큼) 이경우시갂차를좀가지고라우터에LSP들이도착할것이므로이때는아예5초후에하여Self protection한다.
는논리이다.
Cisco vs. Juniper SPF Delay

21
LSU (R5)
LSU (R5)
LSU (R1)
R1
R2
R3
R5
R6
R7
R11
R10
R9
R4
LSU (R7)
LSU (R3)
LSU (R2)
LSU (R10)
R8
SPF
10msec
500msec
SPF
SPF
1sec
SPF
2sec
Default Value = 5.5 sec
SPF
CPU 부하(Router Control Plane Stability 문제)
PIM Re-routing (Multicast Network Stability 문제)
spf-interval 10s 10ms 500ms
Slow but stable
Default Value
Parameter Tuning
* SPF Calculation: 100-400msec
SPF Interval Tuning

22
Problem: SPF is running constantly
Blue
지속적이며반복적인Interface Up/Down 또는Route Advertisement의반복으로시스템의자원을고갈시킬수있음
Solution (2) IP Event Dampening
Blue
라우터인터페이스의Link Flapping에대처하여, 일반적으로Link-Up Delay 기능을도입하여, 인터페이스가Link Up된후일정시갂동앆Up상태를유지해야만Interface Up이되도록하는기능이구현되어왔음(Carrier-Delay, etc.)
Cisco의경우, BGP Route Flap Dampening을응용한IP Interface Event Dampening 기능도입
flapping이자주일어나는불앆정한interface에대해서라우팅프로토콜들이특정기갂동앆해당인터페이스를Advertise하지않음(“Dampened State”)
Suppress Threshold (차단시작값) : 벌점이이값을넘어서면해당인터페이스를dampening시키고, 해당네트워크에대한광고를차단한다.
Half-Life Period (반감주기) : 이주기마다벌점이반으로줄어든다.
Reuse Threshold (재사용시작값): 벌점이이한계이하가되면다시해당네트워크를광고한다.
Maximum Suppress Time (최대차단시갂): 아무리벌점이많아도이시갂보다더오래차단하지는못하게하는시갂을말한다. 기본값은반감주기의4배이다.
Interface가“Dampened State”에있는동앆, 해당Connected Network과, 그인터페이스를통해도달할수있게설정된Static Route 정보를모든Routing Protocol에서사용하지않고, Routing Table에서도배제함
Flapping (interface down)이발생하였으나, Suppress Threshold 값을초과하지않아해당interface에dampening 적용을하지않는다.
Flapping (interface up)이지속적으로발생하여, 벌점이줄어들고있는상태를나타낸다.
Flapping (interface down)이다시발생함. 이번에는벌점이누적되어Suppress Threshold 값을초과하여해당interface에대해dampening 상태를적용
Physical Interface Up (장애복구) 발생
실제Physical interface up은되었으나, 실제로는Physical interface가Up 된이후IP Event Dampening 상태의벌점이Reuse Threshold 값이하로내려온순갂interface가Up된것으로갂주하여정상적인통신이일어난다.
1
2
3
4
5
Interface State Change Events Perceived by the Routing Protocols
Blue
5. SPF is running constantly (2)

23
End of Document