The Vectra S2 appliances offer different challenges during troubleshooting due to the absence of IPMI/iDrac interfaces found on the other platforms. These IPMI/iDrac interfaces permit remote management, troubleshooting and recovery when the primary management interface is unavailable for any reason.
The compact, low profile and low noise nature of the S2 appliances make them suited for small sites and branches, however these environments are also often devoid of the normal management infrastructure present in datacenters making the connection of serial consoles more difficult.
Customers must ensure that all appliances are deployed in environments where access can be arranged for troubleshooting. Vectra requires customers to provide a complete and thorough diagnostic of inaccessible appliances before a replacement will be approved.
Purpose of this article
When an S2 appliance becomes uncontactable there is a natural reaction to assume that the appliance has failed. Replacements are available for all failed appliances under current subscriptions, however in most cases an inaccessible appliance has not failed and therefore a faster and more suitable resolution will be obtained with local diagnostic access.
Vectra aims to assist customers in speedy resolution of all problems and offers this article as a troubleshooting check-list for administrators, users and operators to guide you through the steps to determine whether an appliance has failed.
The troubleshooting steps below should be considered a precursor to raising a support ticket and therefore are also a prerequisite to requesting the replacement of a suspected failed appliance.
There are two aspects to troubleshooting S2 appliances, the troubleshooting which may be undertaken when physically located with the appliance and the troubleshooting which may be undertaken remotely.
- If there are no LEDs lit:
- Are the power supply cords securely attached?
- Is the power button securely in the 'on' position?
- Are the fans running?
- If there are LEDs lit:
- Note down which LEDs are constantly lit, which LEDs are constantly unlit and which LEDs are flashing/intermittent.
- Is the management port connected?
- Are the link lights lit?
- Does the activity light flash?
- If no connection is available check and replace the cable, validate that the cable is connected to the expected switch port and that the switch port is configured correctly.
- Can the device be accessed using the Support port?
- The Support port is an Ethernet port statically configured with the IP address 169.254.0.10/16.
- This port may be used for system installation, configuration or recovery by SSHing to the appliance on the above address.
- Can the device be accessed using the serial console port?
- All appliances should be deployed with known-working console access to allow system recovery or reconfiguration with non-working network connectivity.
- See this article for more details on console access.
- Validate that the serial port you are using is known to be working on another known good device.
- If using a USB-to-serial adapter validate that these adapters are operating normally. Vectra has observed many driver issues with these adapters causing unstable or inoperable serial port connections.
- As a last resort: If you are confident that the device is not responding on the console port then reboot the appliance with the serial console connected. Even in the case of hardware or software failure the S2 appliance will output debugging information on the console and the absence of this debugging information may indicate that the serial port is not working as expected.
- Validate that the IP address of the S2 appliance is as expected.
- Note that S2 appliances use DHCP by default and therefore may obtain a new IP address following a reboot.
- Consider checking the DHCP server to identify whether the device MAC address (obtained from the switch MAC address table) is known to the DHCP server.
- Use console access to confirm the IP address and default gateway using the 'show interface' command.
- Log in to the network device that the device is connected to and validate that:
- The port connection speed is 100M or 1G.
- The port duplex is auto-negotiated and has negotiated full duplex.
- The switch is sending and receiving packets on the port.
- The switch sees the correct MAC address for the device.
- The router sees an appropriate ARP entry for the device.
- The MAC address, ARP entry and expected IP address all match.
- No access control or policy based routing limits the connectivity.
- Validate that the entire network path permits the SSH connection to the appliance:
- Identify access control devices in the path, e.g. firewalls or router ACLs.
- Validate that none of these devices block the connection in either direction.
- Identify that the routing is correct for the source and destination IP in both directions.
- Be aware that ping and/or traceroute may not behave as expected through access control devices and there may not be trusted to give accurate network diagnostic information.
- Capture packets from the network devices at both ends:
- Attempt SSH connection with packet captures running.
- Look for discrepancies in the DNS lookup, ARP process and TCP handshake.
- Validate that the MAC addresses and IP addresses match the expected addresses observed above.
Raising a support ticket
Should you suspect that an inaccessible appliance is faulty you should immediately complete the troubleshooting steps above and raise a Support ticket with Vectra using the web portal (https://support.vectra.ai) or email (firstname.lastname@example.org) including the details and results of all troubleshooting undertaken.