Initial Troubleshooting

• Are a significant portion of cameras offline?
  • Check the Spot Appliances to see if they are offline
  • Check the Spot Appliance to make sure it still has an IP address in the same subnet as the cameras
• Were there any recent network changes?
  • If so, have your network Admin verify the changes didn't affect the Spot appliance or cameras
• Are there only 1-2 cameras offline?
  • Could be cameras out of spec
  • Could be a bad network port or cable
  • Or failing camera hardware

Regardless of whether or not these steps resolve the problem, please open a ticket with Support so that we can perform a root cause analysis.

Step 1: Check the status of your Spot appliances

Navigate to your dashboard and verify that the Spot appliance is still connected. Also, verify if all your cameras are offline or only certain ones.

If the appliance is not connected:

1. Please double-check your connections and reboot the appliance.
2. If a reboot doesn't bring the appliance back online, check the hard drive activity light on the front of the appliance next to the power button. If it is not flickering or blinking at all, reach out to support.
3. More appliance troubleshooting can be found here: Troubleshooting an offline Spot AI Appliance

If the appliance is connected:

Power cycle the POE network switch by unplugging it for about 20 seconds, then plugging it back in.  It should look something like this:

Note: It will usually take the cameras about 5 minutes to show as connected.

Step 2: Check the appliance and camera subnets.

Navigate to your dashboard and verify that the Spot appliance still has access to the camera subnet.

If you're unsure what these are, please contact your Network Administrator.

1. Access Spot IVR's local troubleshooting page (refer to this article for guidance).

2.  Scroll to the "Network Interfaces and Configuration" section at the bottom of the troubleshooting page.

3.  Confirm there is an entry that is in the same subnet as the camera IPs. If there is not, add one.

   • The camera IPs are available under Configure > Edit Location > Cameras

   • The IP address should be in CIDR format, e.g., 172.22.222.5/24, where 172.22.222.5 is the IP address assigned to the IVR and /24 represents the subnet. Contact your Network Administrator for assistance finding an available IP address in your camera's network.

Step 3: Check the camera specs.

• Access the Maintain Page > Cameras, check to see if any cameras are listed as Out of Spec. If there are, you can filter the cameras to only show the Out of Spec cameras by clicking the Out of Spec button.

• Once filtered, hover over the triangle icon to see what is Out of Spec. Please try to adjust any settings listed using the Native Config Page. The Native Config Page is accessible by clicking Configure Camera, by clicking the camera from the Maintain Page, or by accessing the camera from the Configure page for the site. If the camera is fully offline, this step may not be possible.

• In the camera GUI, navigate to Setup > Video & Audio and select the sub-option for "Video"
• The recommended video resolution for cameras to have a better video quality with Spot system should be as followed: 

For Appliances starting with sn5a and up, and enterprise Appliances (snae, snea, snae2):

• Video Encoding: H.265 
• Video resolution: Up to 5MP  (2560x1920 [4:3] or 2880x1620 [16:9])
• FPS: 15
• Rate Control/Bitrate Type: VBR (Variable)
• Bitrate: 2048 Kbps 
• I-Frame Rate: 15 (should match FPS, may not be an available setting)

For Appliances starting with sn4e and up:

• Video Encoding: H.265 
• Video resolution: Up to 2MP  (1920x1080)
• FPS: 15
• Rate Control/Bitrate Type: VBR (Variable)
• Bitrate: 2048 Kbps 
• I-Frame Rate: 15 (should match FPS, may not be an available setting)

Step 4: Check the switch connections.

• Is the camera's cable connection to the switch properly seated? Try disconnecting and reconnecting the cable to confirm it isn't loose.
• Are all the connections on the switch showing as off (no flashing lights) or are just some? If there are no lights at all on the switch, the switch may be dead, or the cameras may not be receiving power.
• If the switch has lights, check to ensure both Activity and Link lights are lighting up for the affected cameras.
  

Step 5: Open a ticket with support

Whether or not your cameras have come back online, please open a ticket with our Spot AI Support Team.

Essential information to provide in the ticket:

• A detailed problem statement.  e.g.:
  "8 cameras offline at Holmes Car Wash at the Baker Street location"
• Troubleshooting performed and the results. If the cameras did come back online, please add that information. 

Optional useful information (Providing as many details as you will speed up the resolution and the analysis):

• The make & model number of your non-Spot equipment (including cameras and networking equipment like your switch). 
• The IP address of one of the offline cameras.
• Are the cameras connected to a DHCP server (e.g. on the same subnet as the router)?
• The login credentials for the cameras.  If you don’t know the login credentials, check whether they are stored on the existing NVR.  
• The DHCP range from your router.
• Provide a diagram of your network to show how the appliance, cameras, switch, and router are connected to each other. 
  Example 1 (flat network):
  
  
  Example 2 (segmented/hierarchical network):
  
  

Advanced Troubleshooting

Physical Network Change

Overview

Could be a network cable that comes unplugged, a switch that gets turned off, or a new switch gets added.

Verifying and Resolving

As this is a physical change, it may be that the only way to verify is to physically trace cables and see if everything is connected and turned on.

• Assuming the appliance is still connected to the Dashboard and is showing as online, start at the camera by verifying it's turned on and connected to the switch.
• Next, verify the network connections going from there to the Appliance (could be through multiple switches or through the same switch). This may be time-consuming depending on the complexity and size of the network. 

If a new switch is introduced and that caused the disconnects, it could be Switch port isolation or a VLAN issue.

Logistical Network Change

Overview

If a switch port had a VLAN configuration changed, port isolation turned on, ACL rules changed.

Verifying and Resolving

• Check the switch ports for any recent changes.
• If there are some cameras online and some offline, compare the switch port configs between the online and offline camera connections to see if anything is different.
• Make sure the Appliance's IP is still able to reach the camera's IP.

Network Cable/Port Failing

Overview

Occasionally, a network cable will go bad, which can cause the camera to go completely offline, or to intermittently go on and off. This is fairly straightforward to troubleshoot.

Verifying and Resolving

To verify this is the case, there are a few ways that may be preferred depending on the network setup:

• Take the problem camera and swap it with a known working camera to see if it comes online.
  • If the problem camera comes online with the new connection, and the known working camera has issues when connected to the problem camera's original connection, the switchport and cable are the suspects.
  • If the issue follows the problem camera, the camera may have gone bad.
• To check if the switchport or cable is the issue, take the affected camera to the switch and connect it with a shorter known working cable. If possible, try to use the same switch port the camera was originally connected to. This allows for the port and cable to be tested separately for further verification of it being a cable or port issue.
  • If the camera comes online and works without issue, the cable is the main suspect, and swapping it out would be the fix.
  • If the camera is still offline or intermittently offline with the new cable, try connecting the camera to a different known working switch port. If it comes online ok, this points to a port issue.

Failing Camera Hardware

Overview

As cameras get older, just as with any hardware, they start to fail and have issues.

Verifying and Resolving

• Verifying can be difficult but there are some indicators that can help. If both port and cable have been verified as being good, as listed in the previous section, yet the camera still has issues, this is one indicator that it is an issue with the camera. 
• Does rebooting the camera bring it back online, only for it to go right back offline after some time?
• To resolve a failing camera, the camera just needs to be replaced.
• Reach out to Support for help with verifying if this is the case.

We hope this article was useful to you, please leave us a comment or feedback as it will help us improve our customer support center.