The photovoltaic (PV) systems' inherent ability to transform solar light directly into electrical energy has contributed to their increasing popularity. However, malfunctions can reduce system dependability. Therefore, rapid hotspot identification is critical for efficient, dependable, and risk-free PV operation. This work presents a method for determining the most optimal hybrid features using the infrared (IR) images of PV panels for hotspot and fault detection. The information at the global (texture, HoG, and color histograms) and local (local binary pattern, SURF, and KAZE) levels were extracted from the IR images of PV panels using a uniform window size of 8 x 8. A binary improved Harris hawks optimization (b-IHHO) optimal feature selection strategy was used to get the optimal feature subset for model training using PV IR images. The IR images of PV were acquired to test the presented framework. The findings suggested that the proposed framework can classify the IR images of solar panels with an accuracy of 98.41% with lesser feature vector size into three classes (normal, hotspot, and defective). Furthermore, the findings were also compared with the latest literature. The presented technique plays a vital role in carbon-free cities and is simple to adopt for PV system inspection.