Systematic literature review for Architectural Approaches in Big Data Systems: Integration of Processing, Security, and Advanced Analytics

Research Questions

The objective of this study is to analyze architectural approaches in Big Data systems, with emphasis on the integration of processing, security, and advanced analytics. To construct the state of the art, the following research questions were formulated:

—RQ0: What are the main trends, approaches, and challenges in the design of architectures for Big Data systems oriented toward the integration of processing, security, and advanced analytics?

—RQ1: How are microservices-based architectures designed and evaluated to support Big Data systems?

—RQ2: What architectural practices and orchestration strategies are used in Big Data systems deployed in container-based environments?

—RQ3: How do architectural decisions influence data management, governance, and integration within Big Data and Data Lakes ecosystems?

—RQ4: What architectural approaches are employed to integrate batch and real-time processing in Big Data systems?

—RQ5: How are security, privacy, and data protection mechanisms incorporated into software architectures for Big Data systems?

—RQ6: How do modern software architectures integrate machine learning and advanced analytics components in Big Data ecosystems?

Search

Table 2. Term, synonyms, and filters to compose the search string. Adapted from [2].

Table 3. Main term and database. Adapted from [2].

Table 4. Search String. Adapted from [2].

Selection

Table 5. Dates of consultation and file download. Adapted from [2].

Table 6. Quality assessment in search and selection processes. Adapted from [2].

Design and evaluation of microservices-based architectures for Big Data systems.

In digital health systems, microservice architectures integrate distributed infrastructures, NoSQL databases, and advanced analytics mechanisms, enabling the application of machine learning algorithms to large datasets for pattern identification and decision support [3]. The adoption of microservices represents an evolution toward decentralized models; however, their systematic integration into Big Data ecosystems remains limited, necessitating reference architectures that articulate modern software engineering with large-scale analytics platforms [4].

Hybrid frameworks that combine microservices with deep learning improve scalability, modularity, and computational efficiency in bioinformatics applications, facilitating real-time processing and interoperability between distributed systems [5]. Reference architectures evaluated using the ATAM method have demonstrated improvements in performance, availability and modifiability, with response times of less than one second and greater capacity to evolve in the face of changes in requirements or volumes of data [4].

Event-driven architectures enable each domain to manage its own information while global consistency is maintained through asynchronous communication, favoring service autonomy and dynamic adaptation of the system [6]. Practical implementation requires container-based infrastructures and platforms such as Kubernetes, which automatically scale services according to demand and maintain the established quality-of-service levels and service-level objectives [7].

Architectural practices and orchestration strategies for Big Data systems in container-based environments.

Microservices deployed in containers managed through Kubernetes allow dynamic scaling of computational resources to maintain quality and service-level objectives, demonstrating that autoscaling optimizes infrastructure under variations in demand [7]. The orchestration of Big Data flows through persistent containers, prioritizing data locality, reduces latency and improves performance, showing that separating the control, data, and compute layers allows the construction of more efficient and scalable platforms [8].

Cloud-native architectures deploy containerized components managed with Docker and Kubernetes, automating deployment and resource management through configuration files that describe the operation of pods within the cluster [9]. Microservices-based collection and analysis systems allow components

such as collectors, download modules, and analysis systems to scale independently within the cluster, improving resilience and the capacity to process large volumes of information [10].

Reactive IoT applications with Big Data analytics apply orchestration patterns such as multi-zone deployments, node affinity, and StatefulSets on Kubernetes, controlling container placement and guaranteeing availability and stable performance [11]. Deep learning frameworks for biological data implement microservices encapsulated in containers orchestrated with Kubernetes and Docker Swarm, executing analyses in parallel within distributed clusters with fault tolerance and elastic scalability [5].

Reference architectures for distributed Big Data employ domain-oriented design with microservices communicating through a service mesh on Kubernetes, using sidecar proxies for load balancing, fault control, and telemetry without modifying application code [4]. Adaptive Big Data platforms deploy autonomous microservices in containerized environments with independent scaling by domain, relying on a service mesh to manage communication and scaling policies that enable dynamic adaptation to workload changes [6].

Health telemonitoring systems structured with microservices in Docker containers managed by Kubernetes allow each component to be replicated and scaled independently, integrating analytics and machine learning services with high availability under increasing demand [3]. Architectures that integrate Big Data and machine learning in distributed environments leverage containerization to partition AI models into parallel tasks executed in pods managed by Kubernetes, reducing execution times through parallelization strategies controlled by the orchestrator [12].

Standard Big Data technologies integrated into containerized environments employ Kubernetes as a middleware platform to schedule Docker containers on cluster nodes, managing resource allocation and facilitating large-scale simulations within distributed infrastructures [13]. Big Data architectures oriented toward advanced services organize containerized microservices in cloud-virtualized infrastructures, enabling modules from different providers to integrate independently and scalably, favoring extensibility and the construction of open data-based ecosystems [14].

Influence of architectural decisions on data management, governance, and integration within Big Data and Data Lake ecosystems.

Domain-based decentralized architectures address critical governance problems through federated control models that distribute responsibility for data, preventing the formation of "data swamps" and facilitating interoperability between services in highly distributed environments [4]. Domain-oriented design with federated governance establishes distributed mechanisms for quality control and metadata management, where each domain retains ownership of its data while global coherence is ensured through asynchronous communication between services [6].

Functional layered architectures manage heterogeneous data through common models and semantic harmonization processes, with a central governance layer that coordinates integration, preprocessing, storage, and querying, eliminating silos and facilitating interoperability [14]. Modern cloud-based architectures decouple storage and compute, automating data pipelines and implementing flexible security and compliance mechanisms compared with traditional monolithic approaches [15].

An adequate Big Data architecture design enables the coherent collection, storage, and processing of information; however, the absence of quality controls in Data Lakes generates inconsistent data that hinders the use of analytical systems [16]. Data Lake architectures with integrated security and monitoring mechanisms strengthen governance by incorporating components that supervise data integrity and lifecycle, facilitating anomaly detection without relying on external solutions [17].

Multilevel governance architectures improve data quality and integration through dynamic rules and traceability mechanisms such as lineage analysis, ensuring consistency and reliability for complex analytical processes [18]. The use of ontology-oriented architectures solves integration problems in heterogeneous environments by providing a common mediated schema that facilitates semantic interoperability among multiple sources within Big Data ecosystems [19].

Layered Data Lake architectures efficiently manage the data lifecycle through cleaning and structuring processes prior to storage, avoiding disorganized environments and facilitating quality and real-time analysis [20]. The integration of blockchain into Big Data architectures strengthens governance through anonymity, access control, and traceability mechanisms, where ETL rules regulate storage and guarantee the integrity of information throughout its lifecycle [21].

Decisions about distributed database architectures influence data availability, consistency, and interoperability, since models based on replication and fragmentation can generate integration difficulties and limit portability between providers [22]. The incorporation of the "mask" element within the mediator-wrapper model improves system flexibility and scalability by redistributing responsibilities among components, facilitating schema management and reducing complexity in integrating heterogeneous sources [23].

Layered architectures in IoT and Big Data environments manage integration through components specialized in acquisition, validation, aggregation, and analysis, guaranteeing data consistency before their use in analytical processes [24]. The relationship between data architecture and governance requires incorporating control, security, and modeling mechanisms from the system design, since the heterogeneity of devices and the distribution of data require structures that ensure the protection and quality of information [25].

Federated learning-based architectures integrate data from multiple entities without compromising privacy, through authentication and encryption mechanisms that guarantee access control and governance without creating dependence on centralized repositories [26]. Layered storage architectures manage data quality through cleaning, standardization, and transformation processes at each level, ensuring that the information is suitable for analysis in Big Data systems [25].

Hybrid architectures that combine distributed storage and blockchain improve governance in Data Lakes through scalable integrity verification mechanisms, where the separation between off-chain storage and on-chain anchoring reduces costs and enables real-time audits [27]. Architectures based on data profiles optimize integration in Data Lakes by automating the discovery of relationships between datasets, facilitating interoperability between heterogeneous schemas and reducing the computational complexity of traditional integration methods [28].

Architectural approaches used to integrate batch and real-time processing in Big Data systems.

The Lambda architecture integrates batch and real-time processing through layers that simultaneously process historical data and continuous streams, ensuring that the results combine persistent information with recent data without interference between both processes [29]. The Lambda architecture divides the system into specialized components that store and process large volumes of historical data while incorporating recent data into the results, achieving a unified and updated view of the system [30].

The Lambda architecture enables the simultaneous analysis of historical and real-time data, proving essential for accurately calculating performance indicators while live data streams are processed [31]. The Lambda architecture as a foundation for Big Data platforms makes it possible to execute batch and streaming processing simultaneously and flexibly, increasing the robustness of the system and its adaptability to different analytical scenarios [32].

The Lambda architecture integrates data from sensors, enterprise systems, and human knowledge, managing them simultaneously, facilitating decision-making in complex industrial environments [33]. The Lambda architecture divides processing into layers that manage historical data and live streams, mapping to pipelines that guarantee continuous and consistent analysis within reference architectures [12].

An architecture that integrates continuous real-time ingestion with periodic batch tasks allows combining immediate analysis with deep processing of large volumes of accumulated data [10]. A data lake architecture that operates in batch, streaming, or combined modes through configurable flows executed simultaneously facilitates the integration of both types of processing within a single system [17].

The use of Apache Spark makes it possible to integrate batch and real-time processing within the same architecture through hybrid models that improve the scalability and processing capacity of Big Data systems [34]. A data synchronization architecture combines batch and real-time processing using differentiated tools for each type of workload, integrating full synchronization with change data capture to keep data continuously updated [18].

The Lambda and Kappa approaches integrate batch and real-time processing by separating or unifying processing layers, enabling the design of systems that handle historical data and continuous streams within distributed environments [4]. The Lambda and Kappa architectures differ in the management of processing layers, serving as a reference for the design of adaptive architectures in Big Data systems [6].

The Lambda and Kappa architectures balance historical and real-time data processing, efficiently managing the data lifecycle and improving the quality of analysis in Big Data systems [16].

Security, privacy, and data protection mechanisms within software architectures for Big Data systems.

A hybrid architecture for Data Lakes integrates cryptographic mechanisms into the data pipeline through authenticated encryption and verification structures based on Merkle and blockchain, processing and validating each record before storage to guarantee integrity, authenticity, and traceability [29]. Modern Big Data architectures integrate security through Kerberos authentication, granular access control with Apache Ranger, and cloud identity management services, directly coupled to storage and processing systems [15].

A secure Data Lake architecture incorporates components dedicated to integrity monitoring as embedded services, integrating anomaly detection systems and checksum controllers that continuously monitor the processes of data loading, processing, and access [17]. A reference architecture incorporates security and privacy as cross-cutting concerns, integrating identity and access management together with policy enforcement mechanisms through sidecars that control access and governance in a federated manner in distributed systems [6].

A blockchain-based architecture for decentralized identity management uses verifiable identifiers to control access and data flow, guaranteeing privacy and traceability through smart contracts over the lifecycle of the information [35]. An architecture with a privacy-oriented analytics layer integrated throughout the entire data flow allows machine learning models to be executed on encrypted and anonymized information using techniques such as differential privacy and federated learning [1].

A Big Data architecture for IoT integrates security components oriented toward identity management, authentication, and authorization through standards such as OAuth2 and XACML, inserted directly into the interaction flow among devices, services, and data platforms [36]. A distributed architecture integrates security through service meshes and sidecar proxies that implement secure communication with mTLS and granular authorization policies, applying consistent access and privacy controls in distributed data environments [4].

An architecture for massive data collection integrates differential privacy directly into the ingestion pipeline, protecting the data from the moment of its capture before being stored or processed [37]. An architecture that integrates encryption, authentication, access control, and blockchain guarantees the security, integrity, and transparency of the data throughout the entire supply chain within a Big Data environment [38].

Integration of modern software architectures, machine learning components, and advanced analytics in Big Data ecosystems.

An Earth system digital twin architecture integrates AI and machine learning components as core modules within information and simulation subsystems, combining physical models with AI-based models according to precision constraints, computational resources, and response time requirements [39]. A Data Lake processing architecture integrates a machine learning model into a hybrid offline-online pipeline that analyzes massive data profiles to infer the quality of joins between attributes and optimizes the execution of large-scale analytical operations [28].

An MLOps-based architecture integrates machine learning components, such as automated pipelines deployed on Kubernetes, in which each stage of preprocessing, modeling, and inference runs as an independently orchestrated pod to support batch processing in Big Data environments [10]. A multilayer architecture encapsulates machine learning and artificial intelligence components by integrating tools such as Spark MLlib and TensorFlow to manage the complete model lifecycle, feeding batch and streaming flows [14].

An extension of Big Data reference architectures incorporates specific modules for the machine learning lifecycle, integrating model development, packaging, deployment, and distribution with AIOps and DevOps practices in distributed infrastructures [40]. An MLOps-oriented architecture integrates advanced analytics through distributed pipelines for model acquisition, processing, and execution, automatically deployed in heterogeneous edge, fog, and cloud environments through standardized containers and inference engines [41].

An architecture based on the Lambda paradigm integrates descriptive, predictive, and prescriptive analytics components at the logical level, combining batch processing for model training with stream processing for generating real-time predictions from industrial data streams [34]. A microservices-based architecture implements generative deep learning models as independent, modular services orchestrated within Big Data workflows to facilitate scalability, updating, and parallel execution of specialized analytical tasks [5].

A collaborative cloud-edge architecture functionally distributes machine learning components, executing inference on edge devices for real-time processing while the cloud manages storage and continuous retraining of models [42]. A Data Lake architecture integrates machine learning models as embedded services within monitoring components, where algorithms such as Isolation Forest, K-means, and convolutional networks perform anomaly detection, clustering, and real-time classification [17].