Deep Multi-Modal Reinforcement Learning-Based Multi-Modal Crop Yield Prediction Network

Abstract

In agriculture, one of the most challenging tasks is predicting crop yield based on different factors, including weather, soil, and crop parameters. To solve this issue, many Deep Learning (DL) models have been developed in the past decades. Among them, the DL-based Multi-Modal Crop Yield prediction Network (DeepMMCropYNet) model takes into account time-series weather, crop, and soil data along with the soil images of specific regions to predict crop yield. This model was built by integrating the Long Short-Term Memory (LSTM)-Temporal Convolutional Networks (TCN) and multi-dimensional Convolutional Neural Networks (CNNs), that support both spatial and temporal feature extraction. However, overlapping data from multiple crops, which can occur in feature space, temporal, and spatial dimensions, limits its performance. These overlaps make it difficult to learn unique patterns for each crop, resulting in inaccurate predictions. Therefore, this article develops the Deep Multi-Modal Reinforcement Learning-based CropYNet (DeepMMRLCropYNet) model by integrating the DeepMMCropYNet with the deep Q-learning (DQL) for crop yield estimation. Initially, the actual output values of the DeepMMCropYNet are mapped into the Q values. After that, the parametric features are integrated with the threshold by the Q-learning agent to forecast crop yield. The agent obtains a consolidated score for its activities by reducing error and enhancing its ability to predict with the best rewarding iterations. Moreover, the total incentives dictate the agents' capacity for learning. Extensive experiments reveal that the DeepMMRLCropYNet achieves a higher efficiency for predicting different crop yields compared to the existing models in terms of Cohen’s Kappa, Mean Square Error (MSE), Mean Squared Logarithmic Error (MSLE), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Correlation Coefficient (R).