This study presents the findings of a systematic review on the use of machine learning (ML) and deep learning (DL) for optimizing user experience in VoIP calls. In order to structure the state of the art, the following questions were posed:

RQ1: How is machine learning transforming defense strategies against fraud and security threats in IP telephony services?

RQ2: What are the most effective artificial intelligence models for predicting user satisfaction (QoE) based on VoIP network behavior?

RQ3: How does AI enable the identification and prioritization of VoIP traffic when it is concealed by encryption and anonymity protocols?

RQ4: How does the integration of artificial intelligence impact massive traffic management to ensure the quality of voice services in SDN networks?

RQ5: How do reinforcement learning and network autonomy contribute to routing optimization and resilience in VoIP systems?

RQ6: What is the role of explainable AI (XAI) in creating VoIP network management systems that are transparent and trustworthy for administrators?

These questions were structured using the PICO model with its PIPOH variant [25], which allows delimiting the essential components of the research to ensure that the findings have technical and professional relevance. Table 1 details these concepts as applied to the study of VoIP network optimization related to machine learning (ML) and deep learning (DL).

Table 1. Definition of general concepts using PIPOH.

Source: own elaboration.

The search strategy consisted of exploring technical terms in specialized databases (IEEE Xplorer, EBSCOhost, DOAJ, ResearchRabbit, OpenKnowledgeMaps). Boolean operators were applied to combine concepts related to IP telephony, learning algorithms, and quality of service and quality of experience. Table 2 describes the terms with their respective synonyms and filters.

Table 3 presents the main topic and the databases that were investigated; in addition, Table 4 includes the string or token constructed from the main topic and synonyms. This token was the fundamental basis for the search and compilation of all the articles that constitute this systematic review.

All studies identified through the search string were subjected to an exhaustive filtering process. Inclusion and exclusion criteria were defined to ensure that the collected evidence was relevant to the impact of ML/DL on the VoIP ecosystem.

Inclusion Criterion (IC): Engineering-focused document that presents the application of artificial intelligence algorithms (ML, DL) for the optimization of security, QoS, or QoE in VoIP networks, published between 2021 and 2026.

It should be noted that, although the chronological criterion prioritizes publications between 2021 and 2026, 8 previous studies (2016–2020) were exceptionally included. These sources were classified as methodological literature, as they were indispensable for defining the software engineering framework [24] and the clinical question structure [25] that govern this review.

Table 2. Terms, synonyms, and filters used to compose the search string.

Source: authors' own elaboration.

Table 3. Main term and databases.

Source: authors' own elaboration.

Table 4. Search string.

Source: authors' own elaboration.

Exclusion criteria (EC): Documents not available for download, or research focused on traditional analog telephony without machine learning components.

The selection was conducted in 3 phases:

Phase 1: Elimination of duplicate articles among the 5 databases.

Phase 2: Discarding non-downloadable articles.

Phase 3: Application of the inclusion/exclusion criteria by reading titles and abstracts.

It was necessary to consult the articles on different dates, as described in Table 5.

Table 5. Dates of consultation and file downloads.

Source: authors' own elaboration.

To ensure the validity of the 25 final documents, 7 quality criteria were applied based on source provenance, relevance of the content regarding VoIP, and the use of machine learning and deep learning. These criteria made it possible to review the titles and abstracts with technical rigor, ensuring that only related research was included.

After the execution of the search strategy, the results were filtered to ensure the quality of the evidence collected. This selection process allowed us to go from 60 initial records to a select sample of 25 relevant studies. The results of this discard flow and the eligibility criteria applied are summarized in Table 6.

Table 6. Quality assessment of the search and selection process.

Source: Authors' own elaboration.

Table 7. Studies included in the systematic review.

Source: Authors' own elaboration.

Regarding proactive defense, the implementation of techniques such as Deep Reinforcement Learning (DRL) enables systems to learn evolving attack patterns. This approach facilitates the mitigation of vishing and data theft in real time through constant monitoring of transitions between periods of voice activity and silence [1].

To combat Telephone Spam (SPIT), nonlinear models using Artificial Neural Networks (ANN) have been developed, which analyze Call Detail Records (CDR) to differentiate between legitimate users and spammers, thereby achieving accuracies above 97% by modeling complex associations among input parameters [2]. Regarding Network and User Behavior Analysis, trust evaluation has been strengthened through the use of social graphs via systems such as ROBO-SPOT, which analyze caller reputation based on the reciprocity and duration of interactions to achieve success rates of 97% in detecting malicious calls [3].

Likewise, the Adaptive Prediction of SIP Signaling allows anticipating attacks through the use of Hidden Markov Models (HMM) and Bayesian inference to detect anomalous message sequences and block malicious signatures even before the call is formally established [4]. Regarding Security in Encrypted Environments and Optimization, among the greatest challenges is visibility in VPN or TOR channels. Deep Learning, through CNN and LSTM models, enables classifying threats by analyzing spatio-temporal characteristics of the flow without the need to decrypt the content [5]. To ensure that these processes do not affect quality of service (QoS), techniques such as State Transition Limitation (LNST) are applied, which reduces

Significantly, the computational load [6].

Additionally, the use of Bootstrap Forest Models and multilayer classification systems has proven to be effective for identifying voice services within HTTPS channels with an F1-Score of 0.90, thus allowing for dispensing with deep packet inspection (DPI) or the use of specific IP addresses and ports for detection [7,9].

The effectiveness of models for predicting satisfaction varies according to the analytical approach, with Stacking Ensembles standing out, such as the QoEPredict model, which combines XGBoost and Random Forest classifiers to achieve 90 % accuracy by learning to resolve conflicts through disagreement features [11].

On the other hand, the use of fuzzy logic through the F-QoS algorithm achieved an accuracy of 93 % (0.93) and had the lowest recorded execution time of 313 µs, substantially outperforming traditional Bayesian methods, which exhibit significantly higher error rates [12,5].

In the field of neural networks (ANN) and SVR, high accuracy is achieved by integrating signal metrics and user identity characteristics, yielding significantly lower errors than classical methods such as PESQ when factors such as gender and accent are considered in subjective evaluation [13]. Likewise, deep learning for time series (RNN-BLSTM) enables prediction of quality-aware traffic with an average accuracy of 97.68 % owing to its ability to analyze long-duration flows of more than 13 hours, thereby facilitating dynamic resource management that is much more efficient than

traditional statistical models [14].

Finally, word recognition (SPQER) uses ML classification to assess the intelligibility percentage, correlating better with human perception (UserMOS) than conventional signal metrics such as PESQ or HASQI, especially in unstable mobile networks with high packet loss [15].

Regarding Identification in Encrypted and Anonymous Environments, visibility in complex channels such as VPN or TOR represents one of the greatest challenges; however, the use of Deep Learning models such as CNN and LSTM allows threats and traffic types to be classified by analyzing the spatio-temporal characteristics of the flow (FSTFs) without the need to decrypt the content [5,9]. This proactive approach allows systems such as the MA-PLE identification engine, based on the learning of hidden representations, to detect VoIP data streams (RTPs) with greater than 99% accuracy from the payload structure, allowing their detection even before signaling information arrives [16].

Likewise, Bootstrap Forest models have proven highly effective for identifying voice services within encrypted HTTPS channels, with an F1-Score of 0.90, which allows administrators to forgo deep packet inspection (DPI) or dependence on specific ports and IP addresses that are usually masked [7,9]. For prioritization and dynamic management, frameworks such as RNN-BLSTM are used, which predict traffic trends for resource allocation [14].

In software-defined networks (SDN), the integration of AI enables proactive management by identifying so-called "Elephant Flows"

(large, long-duration flows), which are the main causes of congestion and quality of service degradation [17].

Deep learning algorithms such as XGBoost, GBM, and GDF implemented in the SDN controller achieve validation accuracies of up to 100 % for predicting these massive flows. Similarly, this capability enables the automatic reservation of optimal routes based on the load of each link in real time [17,18]. This dynamic prediction capability ensures that latency- and bandwidth-sensitive applications, such as VoIP and videoconferencing, maintain guaranteed QoS even under critical fluctuations in massive traffic [17,22].

Reinforcement Learning (RL) allows network systems to continuously learn from their event history, surpassing conventional protocols (such as OLSR or BATMAN) that cannot avoid routes that have exhibited recurring failures or overloads in the past [19]. Improved protocols based on this technique, such as e-RLRP, contribute to resilience through reward mechanisms that optimize the flow path and substantially reduce packet loss and delay [19,20]. Furthermore, network autonomy facilitated by RL enables dynamic adjustments, such as modifying the “Hello” message interval, to compensate for the overhead generated by intelligent management itself. Moreover, the implementation of these models achieves improvements greater than 90% in the voice quality perceived by the end user [19,21].

The fundamental role of Explainable AI (XAI) is to mitigate the "black box" nature of complex deep learning models, making management decisions interpretable and reliable for human operators [17].

Tools such as SHAP (Shapley Additive Explanations) make it possible to identify and quantify which specific attributes (such as packet or byte size) had the greatest influence on congestion prediction or anomaly detection. Consequently, this allows administrators to validate the system’s diagnoses before applying large-scale routing policies [17,22]. Integrating XAI fosters a symbiosis between algorithmic efficiency and human expert judgment, resulting in more transparent and secure VoIP management systems aligned with the organization’s operational intentions [23].