Solar photovoltaics (PV) are susceptible to environmental and operational stresses due to their operation in an open atmosphere. Early detection and treatment of stress prevents hotspots and the total failure of solar panels. In response, the literature has proposed several approaches, each with its own limitations, such as high processing system requirements, large amounts of memory, long execution times, fewer types of faults diagnosed, failure to extract relevant features, and so on. Therefore, this research proposes a fast framework with the least memory and computing system requirements for the six different faults of a solar panel. Infrared thermographs from solar panels are fed into intense and architecturally complex deep convolutional networks capable of differentiating one million images into 1000 classes. Features without backpropagation are calculated to reduce execution time. Afterward, deep features are fed to shallow classifiers due to their fast training time. The proposed approach trains the shallow classifier in approximately 13 s with 95.5% testing accuracy. The approach is validated by manually extracting thermograph features and through the transfer of learned deep neural network approaches in terms of accuracy and speed. The proposed method is also compared with other existing methods.