CLI Reference for Troubleshooting NSX

Quick post documenting some useful CLI commands for troubleshooting NSX, mainly for my own reference. Other useful information can be found at NSX CLI Cheat Sheet and NSX for vSphere Command Line Interface Reference.

ESXi Hosts

Open an SSH session to an ESXi host. The SSH service can be started from the Configure, System, Security Profile page in the vSphere web client, or under Manage, Services when logging into the host UI.

• esxcli software vib list displays installed vibs, add | grep esx to filter.
  
• vmkload_mod -l | grep vd displays the loaded drivers, add | grep nsx to filter, nsx-vdl2, nsx-vdrb, and nsx-vsip kernel modules should be loaded (
  
• /etc/init.d/vShield-Stateful-Firewall status displays the status of user world agent vsfwd which connects the host to NSX Manager.
  
• /etc/init.d/netcpad status displays the status of user world agent netcpa which connects the host to the controller cluster.
  

• tail -f /var/log/netcpa.log tails the user world agent netcpa log.
• Note – to change the logging level for netcpa execute the following commands on the ESXi host:
  • chmod +wt /usr/lib/vmware/netcpa/etc/netcpa.xml gives write permissions to the file.
  • vi /usr/lib/vmware/netcpa/etc/netcpa.xml opens the file in an editor. Find <level>info</level>, press insert to edit the line and replace info with verbose. Press escape twice and enter :wq to save the file and quit.
  • /etc/init.d/netcpad restart restarts netcpad.

• esxcfg-advcfg -g /UserVars/RmqIpAddress lists the IP address of the registered NSX Manager.
• esxcli network ip connection list lists active TCP/IP connections, add | grep 5671 to filter port 5671 used to connect to NSX Manager.
• ping ++netstack=vxlan -d -s 1572 -I vmk3 <VMK> <VTEP> can be used to ping a VTEP IP address using an increased packet size, where <VMK> is the VMkernel to use on the source host, and <VTEP> is the destination VTEP IP address to ping.
  • For example ping ++netstack=vxlan -d -s 1572 -I vmk4 192.168.30.12
  • If the ping comes back successful then we know the MTU is set correctly, since the command specifies a packet size of 1572 (there is a 28 byte overhead = 1600). If the ping drops the packet then try reducing the packet size to 1472: ping ++netstack=vxlan -d -s 1472 -I (again + 28 byte overhead = 1500). If the smaller ping packet is successful but the larger packet is dropped then we know there is an MTU mismatch.
• pktcap-uw can be used for packet capturing, full syntax here.
• esxtop is a useful host troubleshooting tool, type n to switch to network view.

NSX Manager Appliance

Open an SSH session to the NSX Manager. The SSH service can be started from the Summary page of the NSX Manager Virtual Appliance Management page.

• show interface displays information for the NSX Manager management interface.
• show ip route NSX Manager route information.
• show filesystem NSX Manager file system capacity.
• show log manager follow follows the NSX Manager log.

• show controller list all displays the controller nodes status.
• show cluster all displays vSphere clusters managed by the vCenter Server.
• show logical-switch list all displays all logical switch information.
  
• show logical-switch controller master vni 5001 connection displays the hosts connected to segment ID 5001, also replace connection with vtep mac arp.
• show logical-router list all displays all distributed logical router information.