As noted in the previous post, we will explain the logic of IP packet delivery among routers.

Network Topology

Seen in the left figure below is the network topology to help you understand the packet delivery process among routers. In the middle of the figure lies a router (R1), and in the bottom, four servers are connected to the router through the ports ge1/1 ~ ge1/4, respectively. On the top of the router lies two ports (ge2/1 and ge2/2), and two routers (R2 and R3) are connected to R1 through these two ports, respectively. R2 is connected to 100.1.1.0/24 network, and R3 is connected to 200.1.1.0/24 network.

Now, we will see how packets are delivered from SVR1, via R1 and R2, to 100.1.1.0/24 network. The MAC and IP addresses of the routers and servers are as follows:

1. Installing Routing Entries in RIB/FIB Using OSPF Protocol

R1 installs routing entries in the RIB/FIB by performing the following steps:
① R1 exchanges OSPF LSA (routing information) with its OSPF neighbors, R2 and R3.
② It saves all the routing information in the Link State Data Base (LSDB).
③ For OSPF, the best forwarding strategy is "to use the least-cost path". So, it calculates the shortest (least-cost) path to each destination through Shortest Path First (SPF) calculation.
④ It installs the results (the shortest path routing entries) in the RIB.
⑤ Then, the installed routing entries are copied to the FIB on each line card.

In case of the example in the figure above, the following routing information is installed in the RIB/FIB: (1) the destination network 1.1.1.0/24 is directly connected to R1, and the outgoing interface (OIF) is ge1/1, and (2) the destination network 100.1.1.0/24 is not connected to R1, and so the next hop is R2 (20.1.1.1) and the OIF is ge2/1.

Please note that the routing entries for 100.1.1.0/24 and 200.1.1.0/24 are installed in the RIB/FIB according to OSPF calculation, whereas the rest routing entries are installed when the system operator configures an IP address for each interface (port) through CLI (e.g. when configuring an IP address (i.e. 1.1.1.1/24) for interface ge1/1).

2. R1 MAC Address Learning by SVR1 (Filling in ARP Table)

① Now, SVR1 wants to send an IP packet destined to 100.1.1.1 (Destination IP address = 100.1.1.1).
② SVR1 refers to the routing table, and finds out that the destination is matched to the default route (0.0.0.0/0), and thus 1.1.1.1 (R1) is the gateway to use, and lan1 is the OIF.
Note: In case of usual user station (like regular PCs), the gateway IP address for the default route is also obtained when IP address is obtained through DHCP. In case of a server, however, the IP address and the gateway IP address of the default route are manually set by the administrator of each server.
③ SVR1 refers to its ARP table before sending a packet to the destined OIF (lan1) because it should know the MAC address of gateway 1.1.1.1 in order to send the packet. Then, it finds no corresponding entry in the ARP table (ARP miss). 

Note: In Ethernet L2 (Link Layer), the MAC addresses of a sender (SVR1) and receiver (R1) should be respectively entered in the source MAC address and destination MAC address fields of the Ethernet header. 
④ So, SVR1 sends an ARP request packet to lan1 port to get the MAC address of gateway 1.1.1.1.
⑤ R1, upon receiving the ARP request through ge1/1 port, responds to the request by sending an ARP reply packet (informing of a1, the MAC address of gateway 1.1.1.1) back to SVR1.
⑥ SVR1 fills in its ARP table with the received information (saving a1 as the MAC address).

3. Sending a Packet from SVR1 to Router R1 for the First Time
 

① SVR1, now with a1, the MAC address of gateway 1.1.1.1, sends a packet destined to 100.1.1.1 through lan1 port. At this time the packet contains the following information:

[Ethernet Header] Destination MAC address = a1 (R1's MAC address), Source MAC address = m1 (SVR1's MAC address)
[IP Header] Destination IP address = 100.1.1.1, Source IP address = 1.1.1.10 (SVR1's IP address)

② This packet is received by R1 through ge1/1 port of the line card #1, and held in the ingress packet buffer momentarily (during FIB lookup).
③ Next, the packet processor, through FIB lookup (LPM: Longest Prefix Match) for the received packet, finds out the next hop and OIF are 20.1.1.1 and ge2/1, respectively.
④ The packet processor attaches an internal header to the front of the packet, and forwards it to the switch module. This internal header may contain various information depending on each vendor's implementation. However, for easier understanding, here we assume there are only two types of information contained: (1) information about the line card/OIF where the packet is forwarded (i.e. ge2/1 here), and (2) address information of the next hop (router) where the packet is forwarded (i.e. 20.1.1.1).
⑤ Now, the packet is passed through the switch module, and kept in the egress packet buffer of the line card #2.
⑥ The packet processor at the line card #2 checks the ARP table for the MAC address of the next hop 20.1.1.1.
⑦ No corresponding entry in the ARP table. So, the packet processor informs the control module of the ARP miss event (reporting that there is no MAC address for 20.1.1.1).
⑧ In response to the report, the control module sends an ARP request packet to ge2/1 port, and receives an ARP reply packet from R2.
⑨ The control module saves the received MAC address of 20.1.1.1 (i.e. b2) in its ARP table.
⑩ Next, it forwards the saved information to the line card #2 that has informed of the ARP miss event, having the same copied to the ARP table of the card (but, not to other line cards).
⑪ The packet processor at the line card #2 now has the MAC address of the next hop 20.1.1.1 (i.e. b2). So, it can simply forward the packet it previously kept in the egress packet butter to ge2/1. When forwarding the packet, the egress packet processor may either send the packet immediately according to the pre-set QoS policy (scheduling algorithm), or hold it in the egress packet buffer for a while to handle other packet(s) with a higher priority first. The packet forwarded by R1 to R2 consists of the following information.

[Ethernet Header] Destination MAC address = b2 (R2's MAC address), Source MAC address = a5 (R1's MAC address)
[IP Header] Destination IP address = 100.1.1.1, Source IP address = 1.1.1.10 (SVR1's IP address)

Two things to note:
First, the main concern of the line cards is to forward packets at wire-speed. So, other exceptional events like ARP miss are taken care of by the control module.
Second, an ingress packet buffer is where packets are kept momentarily (instantaneously) while FIB lookup is performed, and hence is pretty small-sized (about hundreds of Kbytes). On the other hand, an egress packet buffer is where packets are kept according to the QoS policy when congestion occurs, and hence is pretty large-sized (several Mbytes).

4. Sending a Packet from SVR1 to Router R1 for the Second Time
 

① Now, SVR1 sends a packet destined to 100.1.1 network for the second time.
②③④ and ⑤ are the same as explained above, and so will not be discussed here again.
⑥ The packet processor at the line card #2 refers to the ARP table, and finds out the MAC address of the next hop 20.1.1.1 is b2.
⑦ So, according to the QoS policy (scheduling algorithm), it takes the packet out of the egress packet buffer, and forwards to ge2/1 port. The packet forwarded by R1 to R2 consists of the following information.

[Ethernet Header] Destination MAC address = b2 (R2's MAC address), Source MAC address = a5 (R1's MAC address)
[IP Header] Destination IP address = 100.1.1.1, Source IP address = 1.1.1.10 (SVR1's IP address)