Performance Comparison of Hybrid Machine Learning Models for Predicting Shear Strength of UHPC Beams
Introduction
In the realm of civil engineering, accurate predictions of shear strength are critical for the design and safety of structures, particularly those involving Ultra-High-Performance Concrete (UHPC). In this endeavor, hybrid machine learning models—GOA-XGB (Giant Armadillo Optimization with XGBoost), SHO-XGB (Spotted Hyena Optimization with XGBoost), and LSA-XGB (Leopard Seal Algorithm with XGBoost)—have shown promise. This article explores the performance of these models based on a detailed analysis of their prediction capabilities, reliability, and overall effectiveness.
Model Performance Analysis
The performance metrics for the models are presented in Tables 3 and 4 for the training and testing phases, respectively. Notably, during the training phase, R² scores— which indicate how well the model explains the variability in the data—were exceptionally high. GOA-XGB recorded a score of 0.9941, SHO-XGB scored 0.9943, and LSA-XGB slightly trailed with 0.9912. These values suggest that all three models account for approximately 99% of the variability in shear strength data, affirming their capability to fit the training dataset accurately.
Moreover, the minor differences in R² values, especially for LSA-XGB, indicate a lower propensity for overfitting, a notable characteristic when tested against new data.
Testing Phase Evaluation
As expected, the R² scores decreased during the testing phase, marking a normal drop in performance upon exposure to unseen data. Here, GOA-XGB achieved 0.9570, SHO-XGB secured 0.9594, while LSA-XGB stood out with a remarkable R² score of 0.9802. This suggests that LSA-XGB not only performed well on the training set but also generalized effectively to the testing phase.
The success of LSA-XGB can be attributed to its hyperparameter optimization capabilities. This model strikes an impressive balance between fitting the training data and maintaining complexity, enhancing predictive capacity compared to earlier research in the field.
Error Metrics: RMSE and MAE
Error metrics, namely Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE), further validate the models’ accuracy. In the training phase, RMSE values were significantly low for all models—GOA-XGB at 0.0104, SHO-XGB at 0.0102, and LSA-XGB at 0.0127. Similarly, MAE values indicated minimal errors, with GOA-XGB showing 0.0069 and SHO-XGB 0.0071, while LSA-XGB had 0.0089.
In the testing phase, RMSE and MAE values increased, as anticipated when transitioning to new data. For instance, GOA-XGB recorded an RMSE of 0.0318 and an MAE of 0.0210, while SHO-XGB had an RMSE of 0.0309 and an MAE of 0.0212. LSA-XGB again showcased the best performance with RMSE of 0.0306 and MAE of 0.0208.
Additional Reliability Metrics
Beyond conventional metrics, additional indicators such as the A20 index, W(I), Variance Accounted For (VAF), and U95 were assessed. The A20 index—a measure of the proportion of predictions within a 20% range—remained low in the training set at around 0.0008, demonstrating high accuracy. WI values, close to 1.0, reflected a good conformity with observed values across both training (0.9985) and testing (0.9886) datasets. Noteworthy VAF scores, exceeding 95% across phases, highlighted the models’ effectiveness in reproducing data variability.
The U95 values were low as well, indicating consistent predictions, further supporting the efficiency of all three models in predicting shear strength. While GOA-XGB and SHO-XGB exhibited strong accuracy metrics in the training phase, LSA-XGB emerged as the superior model in the testing phase due to its higher R² scores and reduced error rates.
Visual Assessment of Predictions
Figures 4 and 5 illustrate scatter plots comparing the measured shear strength against the predictions made by the models. These plots effectively demonstrate the models’ predictive accuracy—ideally, points aligning closely with the black diagonal line indicate high correlation between predicted and actual values. In both training (Figure 4) and testing (Figure 5) datasets, most predicted points from all models cluster near the line, confirming their fitting capabilities.
For instance, the GOA-XGB model’s predictions (depicted in red) closely track actual strength values, albeit with minor deviations at higher strength levels. The LSA-XGB model (denoted in blue), however, stands out as the most accurate, following actual values almost perfectly with minimal deviations.
Convergence and Learning Speed
Figure 6 presents the RMSE convergence trends for all three models over iterations. It becomes evident that GOA-XGB achieves the fastest convergence to an RMSE of approximately 74 in a few iterations, quickly learning and optimizing its predictions. Conversely, SHO-XGB, while slower to rise at the beginning, ultimately resorts to a slightly higher and less precise RMSE of about 76.
Interestingly, LSA-XGB portrays a similar convergence pattern as GOA-XGB but attains the lowest RMSE at around 73, showcasing its efficiency in both exploration and refinement strategies.
Model Validation Techniques
Implemented methods like bootstrap residual resampling and tenfold cross-validation further confirm the robustness and generalizability of the models. The residuals from the LSA-XGB model demonstrate normal distribution around zero (Figure 8), indicating absence of severe bias or overfitting.
SHAP Analysis for Model Explainability
Figures 10-12 delve into the interpretability of model predictions using SHAP (SHapley Additive exPlanations) values. A Mean Absolute SHAP Value Plot (Figure 10) ranks model features by their contributions. Here, the Ac feature emerges as the most prominent influencer on model predictions, followed by others like m and pf.
Moreover, scatter plots elucidate the relationships between feature values and their influence on predictions, revealing insights into how each feature steers the model.
Comparisons to Existing Empirical Equations
Lastly, the study juxtaposes the hybrid model predictions against traditional empirical equations for shear capacity, such as Model MC2010 and AFGC-2013. These conventional equations often produce overestimations of shear capacity, potentially compromising structural safety.
The results indicate that while traditional models can fall short in accuracy, LSA-XGB provides a more realistic shear capacity prediction, evidenced by tightly clustered data around a 1:1 ratio in predictive accuracy, exemplified in Figures 15 and 16.
GUI Interface Development
In support of practical applications, a graphical user interface (GUI) has been developed for engineers, promoting user-friendly access to the predictive models discussed. This innovation facilitates real-time parameter adjustment and shear strength estimation, thereby addressing the complexities involved in UHPC beam design.
The comprehensive analyses and visual representation of the hybrid machine learning models’ performance across various metrics showcase their robustness and potential in enhancing the accuracy of shear strength predictions in civil engineering applications.