New metaheuristic algorithms have soared over the past ten years. A common practice in proposing a new algorithm is validating it on several benchmark functions and engineering problems. Although CEC benchmark problems are specifically designed to validate the performance of new meta-heuristics, engineering design problems from pressure vessels to springs have been used in hundreds of papers to prove algorithm efficiency. To date, no benchmark practices have been established yet, i.e., researchers design their own benchmark to validate their algorithm. Thus, the high number of new algorithms combined with the high number of different benchmarking setups complicates the comparing and validating processes. In this paper, we study benchmark practices related to engineering applications. In particular, our exhaustive qualitative systematic review focuses on metaheuristics applied to the tension/compression spring design problem (TCSDP). The aim of this study is threefold: (i) evaluate where the field stands in regards of algorithm performance on the TCSDP, (ii) evaluate benchmarking practices, and (iii) facilitate future algorithm comparison. For these purposes, we first review all the existing metaheuristics applied to the TCSDP in their first publication. For each paper, we gather the data regarding the problem definition, the simulation setup, and the optimized design. We evaluated the data through several metrics to find the best-optimized design so far. Our findings and analysis concluded that the field of metaheuristics and its benchmarking practice have not reached maturity yet. Thus, we recommend some actions to address the issues and provide future research directions.