Internetwork Troubleshooting Guide

Full-Blown Internetwork Troubleshooting Guide​

Warning, This guide requires somewhat deeper knowledge in Internetworking terms.

Let's start...




To simplify things, computer internetwork can be described as layered communication of computer network protocols. The one I use is "Open System Interconnection Reference Model", it divides computer internetworking into seven basic layers. These are the layers:

• Application Layer (DNS, FTP, HTTP, NFS, AFP, SMB/CIFS, SSH, Telnet, etc)
• Presentation Layer (MIME, etc)
• Session Layer (NetBIOS, Named Pipes, etc)
• Transport Layer (TCP, UDP, PPTP, SSL, TLS, SCTP, etc)
• Network Layer (IP, ICMP, IGMP, IPsec, etc)
• Data Link Layer (MAC address, Frame Relay, HDLC, etc)
• Physical Layer (Ethernet, 802.11a/b/g/n, etc)

Now, remember the arrangement of the entries.

When we connect a computer to a network, we work from bottom layers upward. Layers that are related to our troubleshooting will be Physical Layer, Data Link Layer, Network Layer and Transport Layer. Here's the chronological order when you plug a network cable onto your computer:

1. Physical layer:

The physical layer consists of hardware. Several protocols that resides on this layer are: 10BASE-T, 100BASE-T, 1000BASE-T, and many more. In this case we are talking about network interface cards and network interconnect cables (Ethernet, mostly 100BASE-T or 1000BASE-T). When you troubleshoot, first you need to make sure that your cables are OK, every pin on one end are connected to the other pin on the other end (PIN assignment are another matter to discuss). So in a UTP cable, there are 8 pins, each of this pins must connect from one end to the other end of the cable. Next, you need to make sure that the interface is indeed operational, as in the LED indicator blinks when you connect it to a broadband modem/router/switch (simply saying that it's not dead, you can have other ways to check that it's not dead). Once you are sure that both of our "Physical Layer" are OK, we can continue to the next Layer. This layer is responsible for Device-to-Device connection, electrical currents runs on this layer.

2. Data Link Layer

Several protocols that resides on this layer are ARP, MAC, Frame Relay, FDDI, HDLC, IEEE 802.11 (Wifi), CDP, MPLS, and so on. When both of your network interface are online and connected to each other using an interconnect cable, it should start it's own Data Link layer negotiation process. In this case, let's take "Ethernet" as our Data Link Layer example. In Ethernet we got "Media Access Control" sublayer a.k.a MAC address. This is the SECOND layer for our internetwork to be able to connect to each other. In a Managed Switch, you can see the process of each MAC address gets registered in the switch's memory and being processed for link negotiation (10/100/1000 Mbps speed, then is it Full Duplex or Half Duplex). Once the negotiation is done, the network interface should indicate the result (example: if your NIC capable of 100/1000, amber LED = 100Mbps link, green LED = 1000Mbps link). Once the Data Link Layer done it's business, we are connected on the SECOND Layer. The data that "runs" this place is called "frames", so when your Ethernet NIC send something over the network, it's sending "frames" which then get received by another device to be either processed further (a switch, the frame will be "switched" over), or to be received (destination host) . This layer is responsible for Node-to-Node frame delivery, frames runs on this layer.

3. Network Layer

This is where we the user have some degree of control, the previous layers are "auto-initiated". Several protocols that resides on this Layer are IP, ICMP, IPsec, IGMP, and so on. In our case, we use IP, it is responsible for end-to-end packet delivery including routing through multiple internetwork nodes. In IP we get to have "IP address", a unique address that identify our node in the vast IP network. We get to choose the address we want, or if you can't be bothered to make one to use, you can rely on DHCP server to assign an address to you automatically (what is DHCP is beyond my guide).

There are several versions for IP address, but the ones we use are IPv4 (32bit) and IPv6 (128bit). Let's use the IPv4 for now, IPv6 is not common enough to be used/discussed. IP address can be defined by 2 elements, first the IP address number it self (example: 192.168.0.1) and the subnet[work] mask ("255.255.255.0" or can be defined as "/24").

What does it mean? To read an IP address you need both of those numbers.

1. The first part of IP address is the "NETWORK ADDRESS". This is analogous with your "Street address" in your home address. To calculate the network addres is simply "masking" the IP Address with the network address, the result is your Network addres. In my example, 192.168.0.1 subnet 255.255.255.0, will have "192.168.0.0" as it's "NETWORK ADDRESS" (discarding .1 because the mask we use is the "255.255.255").
2. The second part of IP address is the "HOST ADDRESS". This is analogous with your "House number" in your home address. To calculate the host address, you mask the IP address with the rest of subnet mask (the "0" part), which will result a ".1", combine that with the network address, you'd get "192.168.0.1". That's your HOST ADDRESS, 192.168.0.1, with 192.168.0.0 as NETWORK ADDRESS.

For the proper technical way how to calculate IP address, you can find online tools to do that, it is beyond my guide. IP address is self is managed by an organization called IANA (Internet Assigned Numbers Authority). IPv4 are divided into 2 groups, first group is the "Public IP address", the second is "Private IP address". Here's a link to Wikipedia's page about IP addressing, since it's to broad for me to explain...

So, the first rule you need to have a proper IP address, and be able to communicate with other nodes on your network is:

1. You need to have a UNIQUE HOST ADDRESS in that NETWORK ADDRESS.
2. You need to have the same Subnet mask as the rest of the network.
3. You need to be in the SAME network as the rest of the nodes.

Note: 
 
IP address: 192.168.0.1
Subnet Mask: 255.255.255.0
 
This IP address can be written as [I]192.168.0.1/24[/I] (255.255.255.0 = 24bit)

Example: You have 192.168.0.1/24, Your Network address is 192.168.0.0, the Host address is 192.168.0.1. Other Nodes on the network must have the same Network address as your computer, so they must have 192.168.0.2 to 192.168.0.254 as their HOST address, you won't connect to other nodes if those nodes have 192.168.1.2 as their IP address, it's different network than you. Analogous to street address, there are many houses with "01" as the House number, but there are several blocks that has "01"s in them. You need to get the network address right first to be able to communicate with each other.
Connecting this layer to the previous layer, our "packets" in this layer are to be grouped or divided accordingly so that it can be encapsulated (wrapped) into ONE "frame" (Data Link Layer), then transmit the "frame" as series of electrical signals over the network cable to another interface. This layer is responsible for Host-to-Host packet delivery, including to find the best route to get from one Host to another. Packets runs in this layer.

4. Transport Layer

This layer responsible for making sure that our packets reaches the destination host. It can do that by controlling the flow, segment size, the speed of data exchange, etc. For example, if you send an IM to your friend, the TCP will then break up your "IM message" to sizable chunks, encapsulate it to a packet, put some header, on the data, then send it over the IP protocol. When the Packet gets received on the other end, the host on the other end will send your computer an "acknowledged" message, saying that the packet is received intact. Once the TCP on your side received that "ACK", it then continues to send the next package. One IM can take more than one package to be send, so without TCP, you can send "I'll pick you up at 8, dinner at our favorite restaurant" message, but some are lost on the way, making it "I'll pyou u8,doufa ant", which is nonsense. TCP will make sure that it sends the message intact. If an error happens, it will throw an error, so you know that you need to send the message once again.

Test case

Say you have a connection problem between a desktop (connected via ethernet cable) to your broadband router. You can't browse the internet, but the internet is online, your brother can use XBOX Live while you're struggling with your connection.
Here are the steps:

"Physical Layer"

Make sure that your Ethernet adapter is operational, check device manager, look at the back of your case, make sure the LED indicator is blinking. If it doesn't blink, check the cable, change the port on the broadband router to another vacant port, maybe the port is dead or something. Once you determined that the "Physical Layer" is up to snuff, we can go to the next layer.

"Data Link Layer"

Next we check the Data Link Layer, this we need ARP to check it. To invoke your computer to connect to your broadband router, do a ping. Once the ping is done, whatever the result, do a "arp -a" in the command prompt. Look at the bottom of your broadband router, it should list a MAC addres/Physical Addres for it's LAN/WAN interface, look for that MAC Address on the listed Physical addresses thrown by "arp -a". If you can find it, then it is connected on the Data Link layer.

"Network Layer"

Next we check the Network Layer. Of course your broadband router has an IP address, and by default, most of them will have a DHCP server inside them so you don't need to define your own IP address. But let's assume that the DHCP server in your broadband router is demented/crazy/crashed/idiot/whatever, you can't rely on it to get an IP address. Assuming you know the broadband router's LAN IP address, you can makeup your own. Let's say the LAN interface of your broadband router is "192.168.10.1 subnet 255.255.255.0", you can assign your computer "192.168.10.2 subnet 255.255.255.0". Once you hit OK, your interface should have "192.168.10.2" as it's IP address. Do not put anything in the rest of the boxes yet (empty "Default Gateway", "Preferred DNS Server"). Now you can start pinging your broadband router to check the connection.

Case:

1. "Hardware error" reply: You need to check the lower layers again.
2. "Request Timed out", Packet sent 4, Packet received 0, packet lost 100%: All of the ping request somehow got lost in the way, check lower layers.
3. "Request Timed out", Packet sent 4, Packet received 2, packet lost 50%: Some of the ping requests got lost, some made it back, check cable and interface.
4. "Reply from 192.168.10.1...", Packet sent 4, Packet received 4, packet lost 0%: connection is established and are running fine.

"Network Layer: Routing [Default Gateway]"

By this time, your computer is connected to the broadband router, but not yet connected to the Internet. This time you need to put a "Default Gateway" in your computer's TCP/IP properties. Now this one is somewhat tricky part, for this example, the broadband router is the "Gateway", so you can put "192.168.10.1" in the "Default Gateway" text box. For a more elaborate setup, you might need to ask your IT guy/Network Administrator as for what gateway to use. Once you put that number in, your computer should be connected to the Internet BUT WON'T BE ABLE TO BROWSE THE NET JUST YET. To test the connection:
type this in command prompt(Windows):

tracert 208.77.188.193

or type this in terminal (unix/linux/MacOS):

traceroute 208.77.188.193

It should list a number of hosts it passes to reach 208.77.188.193. The first one should be your broadband router, the second is your ISP, and so on.

"Network Layer: Routing Helper [DNS]"

Now when we browse the Internet, we use hostnames, like "www.sevenforums.com". This is an "intermediate form" of IP address that is used in the IP stack. We humans are mostly bad with numbers, so we need something more "verbal" to attach our memories to. For our computer to be able to understand "www.sevenforums.com", we need a DNS server that will resolve that name to an IP address (remember, Network Layer uses IP address, numbers, not some_human_readable_text/hostname).

Now, most of the time in our example, you can put the broadband IP address as the "Preferred DNS server" because most of these devices can act as DNS proxy which will cache the resolved names in it's memory, and forwards the DNS resolution request if it doesn't have the record. For a more elaborate setup, you need to ask your IT guy/Network Administrator for DNS server to use.

Once you put the DNS on it, you can test it by typing this command in the command prompt:

 nslookup www.iana.org

If everything works, it should show you the Address(es). If it fails, it will tell you that it can't contact the DNS server. When that happens, you need to call your ISP/Network Administrator to get the right DNS Server IP address.

Now, if traceroute/tracert shows that your computer's request is being routed through the Internet, and you can resolve hostnames to IP addresses, you are officially connected to the internet.

This concludes my guide. This guide will help you understand the basics of how a computer network works. Some services or antivirus or malware can hinder your computer's ability to connect to some host on the internet, but this is the very basic steps to test if your computer is indeed connected.

I don't talk about Layer 5 and above because it's depends on the OS and applications. Those can take a veryyyyyy loooooooongg time to explain.

Anyway, I'll update this guide sometime in the future if I have time

Good luck with your troubleshooting,

zzz2496

 

[jfar]

Well Done.

 

[zzz2496]

Thanks

 

[Orpheous]

Thank you for taking the time to post this, you've done a good job!

 

[zzz2496]

Thank you

 

[aem]

Take nothing away from zzz's work here, but the Physical Layer, if i'm not mistaken is the physical cable/median for which data is passed through. The ethernet adapter/ports although is a physical device should reside on the MAC (Media Access Control) layer (the lower layer of the Data Link Layer). So the Test case might need abit of rewording.

 

[zzz2496]

aem, can you elaborate?

edit: Physical layer in my mind would be the hardware we can touch, plug, break, smash to pieces D) which include but not limited to: RJ45 jacks, UTP cables, NICs, switches, routers, etc.

 

[aem]

In brief, the Physical Layer in the OSI Model refers to the physical medium with which data (in various forms) travels on. This is best described as cables, hubs and repeaters. Note: Hubs and repeaters are dumb devices with proerties not much different to a physical cable.

Routers are devices which exists on the OSI Network Layer (Layer 3). these are intermediate devices that connects the networks. It's role is to select paths for and direct packets towards their destination. A Process known as routing.

Bridges and Switches exists on the MAC Layer of the Data Link Layer. These essentially build tables of all MAC addresses located on a network segment and map them to associated port.

 

[zzz2496]

Ahh... I see...

How I understand is, the "Physical Layer" is the actual hardware, as in boxes we connect together, cables we plug to each boxes, etc. Make sure that the boxes are on, running normally, interfaces up and running, LED indicators blinking indicating that it is connected to something on the other side, etc (Physical indicators). When talking about the next preceding layer (layer 2/3/4), I'm about the "router" as a "capability based device", it can "route", which works in layer 3.

What if the router's NIC broken down (worn out over time), how exactly can I describe it? A layer 3 device broke? The fact that it's routing capability is still there, just the interface is ruined. How exactly can we "trace" the problem if not from "lower layers" of that device?

A router will always have an interface, let's use ethernet as an example. The interface needs to be checked hardware wise, is it running normally, what does the boot messages say about the interface (on a Cisco router, it will check each of it's NIC upon boot up, and will indicate an error accordingly, either hardware wise or software wise). So, IMHO a router will have layer 1, 2, and 3 all in one device/box. It has a port which we plug UTP cables to it [layer 1], once connected it will start it's MAC auto negotiation [interface speed, half/full duplex connection, MAC address announcement, basically layer 2 stuffs], then when the layer 2 stuffs are up to snuff, it will start to negotiate for layer 3 [DHCP IP address request, etc, layer 3 stuffs]. All that happens in one little box in the corner of the room, when once connected will route your request through the jungle that is known as "Internet"

Not to mention the complexity when you have Cisco routers that has detachable NIC... That'll complicate things even further... I think I'll stand by my wording for now .

zzz2496

 

[zzz2496]

Client <=> Managed Switch <=> LAN Router <==>  Backbone1Router <==> Borderline Router <===> Primary Internet
Client <=> Managed Switch <=> LAN Router<=||                    ||                     ||=> Backup Internet Connection
Client <=> Managed Switch <=> LAN Router<=||                    ||
                                                                ||
Client <=> Managed Switch <=> LAN Router <==>  Backbone2Router<=||
Client <=> Managed Switch <=> LAN Router<=||                              
Client <=> Managed Switch <=> LAN Router<=||

Looking at a diagram like the one I poorly written, it's simply understandable that each device will work as it's intended to work. But the fact behind this simple diagram is that each device has it's own intricacies to handle before it can do real work. Let's dive a bit:

"Switch"

A switch has many interfaces, usually of the same type (on a cheap ethernet switch we usually will see 5, 8, 16, 24 ethernet ports), on a more advanced switch, we might see 2 types or more of interfaces (24 ethernet ports and 2 FC ports for example). A switch by it self is working in Layer 2 functionally... but it has an ethernet port, it has blinking LED indicating that the port is connected. If you plug a cable and the indicator doesn't light up, you will start troubleshooting, naturally will test if the cable is OK, and trying other ports, see if the next port's indicator light up or not. Once the port indicator light up, you [most of the time] will start pinging other hosts, assuming layer 2 stuffs are done automatically (it will negotiate speed/connection type with it's peer (a PC/router/another Switch)). In a router there is a "Physical Layer" work to do, making sure that the cables are OK, interface port is not dead, etc...

"Router"

A router has at least several interfaces, each has it's own addressing/connection method (let's use Ethernet and MAC as layer 2). The router's interface' have to be up and operational electrically (this is layer 1 in a layer 3 classified device). After it's operational electrically, the firmware must be up and running so that it can start negotiating interface speed, half/full duplex, MAC address announced to the layer 2 network, etc. Once all that done, then it can start it's Layer 3 work.

This is how I "view" computer internetworking, reading a book/diagrams will sometimes makes us forgot the actual "work".

zzz2496

 

[Barman58]

What we have to remember here is that the OSI model is exactly that - a model. A physical piece of equipment can span more than one layer in the model, so a router can exist in both the physical and data link layers

 

[aem]

The teaching of networking will be best left to the experts. I'm like yourself know what we know, from what we read here and there so if i am wrong, then i am wrong lol.

Ok, lets see how far i can go....

What if the router's NIC broken down (worn out over time), how exactly can I describe it? A layer 3 device broke? The fact that it's routing capability is still there, just the interface is ruined. How exactly can we "trace" the problem if not from "lower layers" of that device?

You wouldn't say a Layer 3 "device" broke plainly like that. Routers work off Layer 3 protocols. Each layer is connected to one another logically and information must travel through each layer down the order untill it reaches the physical layer where it then leaves your machine and heads towards the physical router. The information will go from the machine's interface port onto the medium (cable) to the router's interfact port. If say a ping of the router fails, the data packet will know this and it will either cease to exist or back track. Information then goes in reverse back to your machine, through the same interface, and up the layers. You won't know that a ping has failed untill the information reaches a layer that processes this information. Such activities when you break it down, might seem long, but in computer terms is can be as quick as a flash. When the machine gives you the info that a ping has failed, say host unreachable or request time out, you would then know that the ping has failed, and it failed not at physical level, but at a network level. Obviously if a ping fails at a network level (logical level) there is a good chance there might be something wrong with the router or an interface of the router, as the physical level.

 

[aem]

Each machine work off the OSI model with protocols inter-linking from layer to layer on the machine. Information travels from Layer 7 down to Layer 2's logical link layer, and when it reahes the MAC layer of Layer 2, this is when logical becomes physical.

A router is a device/hardware seperate to the machine. Information goes from the machine to the router "physically" (thru a medium, and yes as hard as it may sounds wireless is also a form of physical connection) but the informatin of say a ping failure gets processed at the network layer. This is why the router is known to be a layer 3 device, as much as a hardware.

 

[zzz2496]

The teaching of networking will be best left to the experts. I'm like yourself know what we know, from what we read here and there so if i am wrong, then i am wrong lol.

Ok, lets see how far i can go....

What if the router's NIC broken down (worn out over time), how exactly can I describe it? A layer 3 device broke? The fact that it's routing capability is still there, just the interface is ruined. How exactly can we "trace" the problem if not from "lower layers" of that device?

You wouldn't say a Layer 3 "device" broke plainly like that. Routers work off Layer 3 protocols. Each layer is connected to one another logically and information must travel through each layer down the order untill it reaches the physical layer where it then leaves your machine and heads towards the physical router. The information will go from the machine's interface port onto the medium (cable) to the router's interfact port. If say a ping of the router fails, the data packet will know this and it will either cease to exist or back track. Information then goes in reverse back to your machine, through the same interface, and up the layers. You won't know that a ping has failed untill the information reaches a layer that processes this information. Such activities when you break it down, might seem long, but in computer terms is can be as quick as a flash. When the machine gives you the info that a ping has failed, say host unreachable or request time out, you would then know that the ping has failed, and it failed not at physical level, but at a network level. Obviously if a ping fails at a network level (logical level) there is a good chance there might be something wrong with the router or an interface of the router, as the physical level.

In my days in Cisco training lab (it was 2002/2003 iirc), when I encounter such problem I'm told to (according to my teacher's words) "trace from lower layers upward", this by his definition is make sure what you can touch, plug, etc is up and running, no diagnostic LED blinking, check cables, switch cables if you're in doubt, check that the plug is not loose, replug if loose, etc. This "trick" helped me in my 6 professional years as "All around network handyman".

I won't even start to guess "maybe it's driver, maybe it's X, maybe it's Y, maybe it's Z"... I'll start from the bottom (make sure the connectors are tight, LED indicator light up, etc), then go up (check ARP, ping, recheck ARP), then after that check the IP (ping), after that check the layer 4 (make sure that the ports are open and is listening to requests), and so on...

This is my "way" of doing things, I'm sorry if my way of thinking is not "in harmony" with yours .

zzz2496

 

[aem]

The test case is all well except you might want to put in a Media Access Control Layer as part of the Data Link Later, at this half of Layer 2, it's where you talk about the adapters. The physical Layer 1, is where you say something like make sure you have the correct cable and it's inserted properly. It's up to you but for a non tech person that not necessary i don't think.

 

[zzz2496]

Each machine work off the OSI model with protocols inter-linking from layer to layer on the machine. Information travels from Layer 7 down to Layer 2's logical link layer, and when it reahes the MAC layer of Layer 2, this is when logical becomes physical.

A router is a device/hardware seperate to the machine. Information goes from the machine to the router "physically" (thru a medium, and yes as hard as it may sounds wireless is also a form of physical connection) but the informatin of say a ping failure gets processed at the network layer. This is why the router is known to be a layer 3 device, as much as a hardware.

In my mind, when I experience ping failure, I instinctively will start from checking physical connections (in my word ==> "physical layer"), going up to second layer (MAC address, check this with ARP command), and so on... This way I know in what "layer" it fails and then act accordingly. I don't quite care about guessing why a ping command fails, it's irrelevant... I'll just trace it, by the time I arrive at layer 3(IP), most of the time I already know what the culprit was...

Note: In my days, I use many other link layer (layer 2) protocols other than Ethernet/MAC, I once setup Frame Relay connection (that's a Layer 2) using several Cisco routers over serial connection (thick heavy duty cables, Cisco proprietary), and it uses HDLC frames, at layer 3 uses IP. Another time I got a client that's running over microwave, I forgot what the Link layer protocol was... At that time, if I were think as if this network stack is one huge complicated mess, troubleshooting a failed ping will take days, even weeks... . That will definitely get my arse fired...

zzz2496

 

[zzz2496]

We are commenting each other's post in minutes intervals...

zzz2496

 

[CarlTR6]

ZZZ, this is outstanding work!

 

[zzz2496]

After reading this guide, I suggest you read the other one...

zzz2496

 

[CarlTR6]

I did! And is is excellent, too. I repped you on that one. Unfortunately, I can't rep you again.