Wireless troubleshooting can encompass a lot of variables.  There's the wired network.  There's the client supplicant configuration.  There's client drivers.  There's the authentication server.  There's the wireless hardware.  There's the wireless firmware.  There's external WiFi factors.  There's external non WiFi factors.  There maybe configuration at multiple points.  There's the physical layout of the APs themselves.  And many more could surely be listed. And still a lot of the time wireless can be pretty forgiving and just work well enough to not be resolved. 

With everything WiFi troubleshooting encompasses I was recently reminded of some troubleshooting basics.  It was a simple and straightforward issue: an AP at a new site that wouldn't join to the controller.  Everything that could have been suspect had been looked at: VLAN, routing, DNS, DHCP, swapping in new AP hardware, etc.  By all appearances everything should've just worked.  The AP would power up just fine, the controller was reachable by ping, their wasn't a firewall port stopping traffic, the controller GUI was working.  Having discovered what it wasn't narrowed down what it might be.  A few more minutes of troubleshooting led to the diagnosis:  a bad ethernet cable.  The cable was providing power to the AP, but the data pairs weren't passing traffic.  A simple solution for what had been a time consuming problem.

What can help in troubleshooting problematic scenarios?  I've found the CWNP troubleshooting methodology as described in the Certified Wireless Analysis book to be very helpful.  This system has been designed to look at issues in a logical and systematic procedure.  In summary the procedure is described as:

1. Identify the problem.
2. Discover the scale of the problem.
3. Define the possible causes of the problem.
4. Narrow to the most likely cause.
5. Create a plan of action or escalate the problem.
6. Perform corrective actions.
7. Verify the solution.
8. Document the results.

While not every problem is easy to solve, having the right tools and the right mindset can go a long way.  Develop a game plan and work it until the problem is resolved.  It might be simple or complex but there's going to be a solution.       
        

WLAN Pi

I've recently begun exploring the capabilities that can be found in the WLAN Pi.  The WLAN Pi is a compact network tool designed to test wired-wired, wired-wireless and wireless-wireless networks.  The tools that you get in such a small and portable size are great for quick troubleshooting and analysis.  The small size makes it ultra portable so it's very easy to pack around full time when you might not be able to do that with other tools.  For full details on the hardware setup head over to WLANPI.com

Under the hood the WLAN Pi currently comes installed with a number of tools including:

Iperf3 server 3.1.3
Iperf2 server 2.0.9
ZAPd (Ruckus) daemon 1.83.18
Ekahau eperf
Speed Tests (HTML5)
Kismet Dev build
WiFi Explorer Pro Sensor Works with WiFi Explorer Pro (macOS only)
H.O.R.S.T - Live Wi-Fi packet analysis
TCPDump - common packet analyzer that runs under the command line
HOSTAPD - IEEE 802.11 AP and IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator

Apps like Speed Test, Speed Graph, Kismet and Kismet Mobile are GUI accessible and utilize a simple one button start to run operation.

Speed Test

Kismet Wired Speed Test

Kismet Wireless Speed Test


 

Speed Graph

Kismet

Kismet offers more than simple push button operation.  From the Release Notes:

Kismet is a wireless network detector, sniffer, and intrusion detection system. Kismet works predominantly with Wi-Fi (IEEE 802.11) networks, but can be expanded via plug-ins to handle other network types.
● 802.11 sniffing
● Standard PCAP logging (compatible with Wireshark, TCPDump, etc)
● Client/Server modular architecture
● Plug-in architecture to expand core features
● Multiple capture source support
● Live export of packets to other tools via tun/tap virtual interfaces
● Distributed remote sniffing via lightweight remote capture

Kismet's scanning utility wifi network summary will list the MAC address/SSID, signal strength, channel, number of detected clients, occupied channels, among other information.

Drilling down into the details:

HORST -- Highly Optimized Radio Scanning Tool

From the Release Notes:  “horst” is a small, lightweight IEEE802.11 wireless LAN analyzer with a text interface. Its basic function is similar to tcpdump, Wireshark or Kismet, but it’s much smaller and shows different, aggregated information which is not easily available from other tools. It is mainly targeted at debugging wireless LANs with a focus on ad-hoc (IBSS) mode in larger mesh networks. It can be useful to get a quick overview of what’s going on on all wireless LAN channels and to identify problems.

Features:
● Shows signal (RSSI) values per station
● Calculates channel utilization (“usage”) by adding up the amount of time the packets actually occupy the medium
● “Spectrum Analyzer” shows signal levels and usage per channel
● Graphical packet history, with signal, packet type and physical rate
● Shows all stations per ESSID and the live TSF per node as it is counting
● Detects IBSS “splits” (same ESSID but different BSSID – this is a common driver problem)
● Statistics of packets/bytes per physical rate and per packet type
● Has some support for mesh protocols (OLSR and batman)
● Can filter specific packet types, source addresses or BSSIDs
● Client/server support for monitoring on remote nodes
More details: https://github.com/br101/horst
Developer: Bruno Randolf - @spiralsun69

Missing APs

The Cisco WLC is great at many services, but providing historical information is not one of them.  Cisco would refer you to Prime Infrastructure or DNA Center for historical WLC, client or AP information.  And collecting data is the strong point of these other products.  The Reports section over at Prime Infrastructure is loaded with all sorts of archived information and statistics and the landing page broadcasts loudly where potential trouble spots are at.  But what if you don't have easy access to Prime or DNA Center?  One useful feature if you manage hundreds of Access Points is the AP Join information found under the Statistics category.  Here the last known join time of an AP is recorded as well as its current status of Joined/Not joined.  If you start seeing complaints about poor wireless or for just a quick health checkup of the Access Points this is a great place to check out right from the WLC homepage.

Troubleshooting an AP by the LED

When you find yourself troubleshooting an AP it can sometimes be made a little easier with a quick glance at the LED light.  It might not always be easy especially when an AP has been mounted at some height, but it can give some insight even when you don't have any tools or network access.  Summarized here are some useful states you might encounter on a Cisco Access Point.

Blinking Green: AP startup
Solid Blue: Client is associated.
Solid Green: AP is functioning normally, although without any clients associated to it.
Blinking Blue: Firmware upgrade is in occurring.
Cycling Red-Green-Off: WLC Discovery/Join
Cycling through Blue-Red-Green-Off: General warning; insufficient inline power

On a Meraki Access Point these are the status LEDs.

Orange: AP startup
Blinking Blue: Firmware is upgrading
Green, Blue, Purple, Red: Establishing session to the cloud controller
Green:  AP in Gateway mode with no clients associated
Blue:  AP in Gateway mode with clients associated
Blinking Orange:  AP can’t find an uplink and may have an internet connection issue

Two other helpful LED light options are available from the Controller.  You can disable the LED completely or if you're trying to identify an Access Point you can send a command to blink the LED.  This can be done from the Advanced tab per AP or from the CLI.
         

 

(WLC) >config ap led-state ?

              
disable        Disables the LED-State for an AP
enable         Enables the LED-State for an AP
flash          Configure the LED-flash for an AP
              
(WLC) >config ap led-state flash ?
              
<seconds>      The duration of LED flashing in seconds <1-3600>
              
indefinite     Indefinite flashing of the LED
              
disable        Stop flashing the LED

Meraki APs also have "dark mode" where the LEDs are disabled.  This option is configured under Network Wide- General- LED Lights.

Changing Cisco AP Image Versions

While a Cisco Wireless Controller will keep the Access Point firmware in sync, you may need to swap code between Autonomous or Mobility Express images.  This post will explore three common scenarios and how to do that.

First let's identify the code we need to install.  With a quick glance at the filename we can determine if the image is designated for a lightweight, autonomous or Mobility Express installation.

For a lightweight code image look for the k9w8 tag.  ap3g1-rcvk9w8-tar.153-3.JF8.tar
For an autonomous image look for the k9w7 tag.  ap3g1-k9w7-tar.153-3.JF8.tar
For a Mobility Express Image look for the ME tag.   AIR-AP3800-K9-ME-8-8-100-0.tar

Lightweight to Autonomous

I've found the most convenient way to convert an AP from lightweight to Autonomous mode is by using a Cisco Controller and TFTP server.  This can be accomplished quickly and easily with just a single command.  Note that this procedure is for pre-Wave 2 AC APs only, as they support the Autonomous image.

From the WLC CLI interface:

config ap tftp-downgrade tftp-server-ip-address filename access-point-name

(WLC) >config ap tftp-downgrade 10.10.10.5 ap3g2-k9w7-tar.153-3.JF8.tar 3502i

The WLC will send the command to the AP and the AP will download the code, reboot and come up in Autonomous mode.

Autonomous to Lightweight

To move the AP from autonomous to lightweight mode, we can execute a single command.

archive download-sw /overwrite /reload tftp://IPADDRESS/lightweightcodename.tar

The AP will download the image and automatically reboot.

Convert To Mobility Express

If you have a lightweight AP that you want to convert to Mobility Express follow these steps.

From the console login to the AP.  Note that if the AP has previously joined to a Wireless Controller the default login may be disabled.  If the default username/password doesn't work verify from the WLC Global or individual AP credential option.

At the AP CLI initiate the code transfer with the following command:

ap-type mobility-express tftp://<tftp server ip-address>/<filename of TAR file with path from root on the TFTP server>

For example:  ap-type mobility-express tftp://10.10.10.10/AIR-AP3800-K9-ME-8-8-100-0.tar

The AP will reboot and load the Mobility Express image.

Mobility Express to Lightweight

If you have a Mobility Express deployment and want to move all of the APs over to a lightweight install this can be accomplished with a single command.   First, let's verify our Mobility Express APs from the CLI.

(Cisco Controller) >show ap summary

Number of APs.................................... 2


AP Name                         Slots  AP Model              Ethernet MAC       Location              Country     IP Address       Clients  DSE Location 
------------------------------  -----  --------------------  -----------------  --------------------  ----------  ---------------  -------  --------------
*3802                            3      AIR-AP3802I-B-K9     
1832                            2      AIR-AP1832I-B-K9 

Having verified the APs joined to the WLC are the ones we want to convert, we can now initiate the command.
 
(Cisco Controller) > config ap unifiedmode <switch_name> <switch_ip_address>

>config ap unifiedmode WLC 192.168.1.2
Warning! Sending config unifiedmode to all APs will cause the WLC to be rebooted.
 Are you sure you want to continue? (y/n) y

The APs will reboot and join the Controller as programmed in the command.

Cisco WLC Code-- What Should I Be Running?

Two questions I'm frequently asked are "What version of wireless code should I be running?" and "Is there code support for X hardware with Y software?"  Although the answer will vary depending on the specific hardware and features required, Cisco has provided two very helpful resources that should be consulted.   

For recommended AireOS/Mobility Express code releases the Cisco TAC Recommended AireOS Builds is a great place to see what TAC currently judges as the best software release

After reviewing what TAC recommends, be sure to always check the Release Notes for each version of code for a detailed breakdown of what's in each release, including What's New, supported/unsupported features per platform, and Resolved or Unresolved Caveats.

With so many AP models being produced in the last few years maintaining compatibility between WLC software and AP hardware can be a challenge.   To quickly see what software support is required for a specific model of AP or when software support ended, what version of Cisco Prime is compatible with a WLC, CMX, or MSE release, the Cisco Wireless Solutions Software Compatibility Matrix is the place to go.  

Cisco Mobility Express

Cisco's Mobility Express solution seeks to simplify the management and speed the process of creating a wireless network without utilizing a traditional Wireless LAN Controller.  Intended for small business and supporting a maximum of 50-100 access points depending on which model AP acts as the Master AP, its goal is to bring the automated intelligence features associated with traditional wireless controllers directly onto the access point platform.

The Mobility Express platform may be a suitable replacement for administrators supporting traditional Autonomous versions of Cisco Access Points, however always check the Release Notes to see if a specific feature is supported since Mobility Express is not designed to provide feature parity with traditional controllers.  Also note that autonomous code is no longer being released for any Cisco Wave 2 AC Access Point.

This post will explore the initial configuration of Mobility Express platform.

There are two ways to access the Mobility Express setup, traditional CLI through the console port and wirelessly using the setup specific CiscoAirProvision SSID.  Whether using the CLI script or the CiscoAirProvision SSID initial setup can be completed in a matter of minutes.  Upon completion of the setup script, the access point will have been assigned its network information, including the static IP address you will use to access the management interface, and the optional initial SSID setup for both a corporate and guest wireless network and their networks.

Would you like to terminate autoinstall? [yes]:
Enter Administrative User Name (24 characters max):
Enter Administrative Password (3 to 24 characters): 
Re-enter Administrative Password                 :

System Name [Cisco_1e:ab:c0] (31 characters max):

Enter Country Code list (enter 'help' for a list of countries) [US]: us

Configure a NTP server now? [YES][no]: yes
Use default NTP servers [YES][no]: yes

Enter timezone location index (enter 'help' for a list of timezones):

Management Interface IP Address: x.x.x.x
Management Interface Netmask: x.x.x.x
Management Interface Default Router: x.x.x.x
Cleaning up Provisioning SSID
Create Management DHCP Scope? [yes][NO]: x
Create Employee Network? [YES][no]: x
Employee Network Name (SSID)?: x
Employee VLAN Identifier? [MGMT][1-4095]: x
Employee Network Security? [PSK][enterprise]: x
Employee PSK Passphrase (8-38 characters)?: x
Re-enter Employee PSK Passphrase: x
Create Employee DHCP Scope? [yes][NO]: x
Create Guest Network? [yes][NO]: x
Enable RF Parameter Optimization? [YES][no]: x
Client Density [TYPICAL][Low][High]: x
Traffic with Voice [NO][Yes]: x

Configuration correct? If yes, system will save it and reset. [yes][NO]:

If you choose to use the CiscoAirProvision option the same initial features are available using the GUI.  The CiscoAirProvision SSID is Open access and is removed after the initial configuration is done and the AP is reset.

The initial configuration of the Mobility Express solution is very streamlined and can be accomplished in just a few minutes.