We applied the 9 domains of MAS-AI, illustrated in Fig. 2, to the BLOC-OP technology, fulfilling a full MAS-AI assessment comprehensive of: the detailed description of each domain, the answers to the 5 “process factors”, and, finally, a summarizing table of the assessment.

Fig. 2

Graphical representation of the MAS-AI framework, illustrating its nine domains [15]

Domain 1: The health problem and current use of technology

The health problem related to operating block management involves a significant margin of error due to planning based on operators’ personal experience. Procedures in the hospital involve weekly meetings between surgeons and anaesthesiologists to validate scheduling, mostly based on the average duration of surgery, but this can lead to improper use of operating rooms [18]. To address this problem, the use of artificial intelligence and machine learning techniques is emerging as a promising solution. Recent studies have shown that the application of machine learning models can improve the scheduling accuracy of surgical operations and reduce delays, thereby optimising resource allocation [19].

However, the effective application of such models requires data of high quality and accuracy. Today, in Parma’s University Hospital, the recording of patient times and movements in the operating block is done manually, via a information system called “Ormaweb”. The use of an automated patient tracking system could improve data quality and reduce human errors. An autonomous registration system could also lighten the workload of the operators and reduce the changeover time between patients. A positive example is the experience of the hospital in Cesena, an Italian city, where the implementation of aRadio Frequency Identification (RFID)- based tracking system has improved the accuracy and completeness of trauma patient documentation [20]. We expect that the application of AI could elaborate all the data coming from an automated patient tracking system, while taking into account all the variables that affect the succession of surgeries. This could avoid the errors of manual planning in the surgery and lead to more effective clinical and organisational management of the operating theatre.

Domain 2: Technology

The authors of the MAS-AI argue that the technological aspects of the evaluation can be divided into two main categories: those related to the development of the technology and those related to its implementation in the specific clinical context being evaluated. In this case, the BLOC-OP protocol is mainly concerned with the former, so our focus will be more on evaluating those. Furthermore, Domain 2 of the MAS-AI refers to the CLAIM (Checklist for AI in Medical Imaging) [17] guidelines, which are not applicable in this case because BLOC-OP is an AI system but is not based on imaging. Therefore, we will limit ourselves to framing the technological aspects according to the description provided in the BLOC-OP’s project protocol. In terms of technology development, the BLOC-OP project consists of two phases. The first phase consists of the systematic collection of data from the central operating block, the Recovery Room and the on-site monitoring of vital parameters. During this phase, data will be collected on operating room times, patient biographical and anamnestic data, data from the Recovery Room and data on hospital stay.

In order to ensure the traceability of patients, a system consisting of a Bluetooth tag associated with the patient’s identification code will be used, which will be placed on the medical record when entering the operating theatre. The operating theatres and the Recovery Room will be equipped with Raspberry Pi systems capable of detecting and storing the movements of the tag within the wards. The data recorded by the Raspberry Pi systems will be periodically extracted via Bluetooth technology using dedicated tablets. The second phase of the project involves the application of calculation techniques to develop a technology capable of providing an optimised organisation tool for the operating block. During this phase, Machine Learning techniques may be used to solve complex problems where it is not possible to define specific instructions or rules to achieve an optimal solution. The BLOC-OP is based on the analysis of a large dataset, which is used to build a predictive analysis system. To enable the discovery of hidden patterns, a large number of data must be analysed. In this case, approximately 50 variables were identified for each patient, which will constitute a single instance of the dataset.

The size of the dataset is compatible with a 6-month data collection campaign for a total of 1,200 patients. The dataset was divided into three parts:

1.

Training set (60%): This data set is used to train the algorithm and build the model. During this phase, the algorithm learned the correlations in the data.

2.

Validation set (20%): This validation set is used to calculate the accuracy or error of the classifier and to remodel parameters based on the results obtained. Instances of the validation set are not used during model creation or optimisation.

3.

Test set (20%): This data set represents the future data that will be analysed. It is used in a similar way to the validation set but was never used during the creation or optimisation of the model.

The development of the BLOC-OP AI technology follows three basic steps:

1.

Data understanding: During this phase, relevant data are identified to create specific models governing the different parts of the complex phenomenon being examined. A univariate analysis is performed to examine the characteristics of each variable in the data set. This process helps to better understand the distribution and relationships in the data.

2.

Data preparation: In this phase, data identified as relevant are processed using feature engineering. Feature engineering consists of transforming the data into other variables suitable for use as input in predictive models. During this phase, data normalisation, a process that rescales the variables so that they are comparable on the same scale, preventing some variables from having too much weight compared to others, may be required.

3.

Modelling and Evaluation: In this phase, the most suitable algorithms for the problem are identified and the training and validation of the model created by the selected algorithm is carried out. Depending on the input variables identified in the previous phase, different modelling techniques can be used. For example, if the relationships between the data are clear, a Bayes classifier or a Decision Tree, which are algorithms based on probability calculation, could be used. Conversely, if the relationships are more complex, neural networks and deep learning can be used for modelling.

Although the BLOC-OP system is not imaging-based and the CLAIM guidelines were developed specifically for medical imaging AI tools, several relevant principles from CLAIM can be extrapolated [21]. In particular, the need for reproducibility, technical transparency, and performance validation are highly pertinent. During development, one of the main challenges was integrating heterogeneous data sources—including surgical schedules, preoperative information, and intraoperative timings, into a unified and structured dataset. Ensuring data quality required custom pipelines for data cleaning, synchronization, and validation [10].

Another technical hurdle involved the traceability and reliability of Bluetooth-based patient tracking, which had to be tested under variable physical conditions within the operating block. Close collaboration with clinical engineers was essential to ensure compatibility with existing hospital IT infrastructure and to avoid interference with Ormaweb/O4C systems. Furthermore, ensuring compliance with data protection regulations such as the GDPR demanded early anonymization strategies and strict data access protocols [22]. Although the project predated specific AI ethical frameworks, an emphasis was placed on human oversight, data security, and explainability throughout the design.

Domain 3: Ethical aspects

The project team of the BLOC-OP project assure that the study was conducted in accordance with Good Clinical Practice (GCP) standards and the Declaration of Helsinki [23]. Both documents represent an international standard of ethics and quality required for the design, conduct, recording and reporting of clinical trials involving human subjects. Compliance with them ensures compliance with basic ethical principles such as the protection of the rights, safety and well-being of research participants, and fairness in the assessment of risks and benefits. However, the authors do not explicitly refer to ethical aspects that specifically concern AI systems.

In fact, current AI-based clinical decision support systems (CDSSs) raise significant ethical challenges that extend beyond traditional research ethics frameworks. These include the delegation of decision-making processes to non-transparent, “black box” algorithms, which can obscure responsibility attribution in the event of diagnostic or organisational errors, and compromise clinicians’ autonomy and accountability. The lack of explainability also poses a barrier to trust, informed consent, and contestability of automated outputs. As highlighted by Bertl et al. (2023), these unresolved issues remain among the main barriers to the clinical adoption of AI technologies in healthcare decision-making processes [24, 25].

As a guarantee of transparency, the project allows the Ethics Committee to monitor, verify, review and inspect data and original documents, in order to ensure compliance with ethical standards and regulations. However, it should be noted that the BLOC-OP project was conceived before the publication of specific ethical guidelines on the development and application of AI in healthcare.

Domain 4: Legal aspects

During the development of BLOC-OP lawyers with experience in the AI regulation field were consulted to ensure compliance with the regulatory framework and to adapt the informed consent forms. Patients were in fact invited to participate in the study through informed consent, and only included once consent has been obtained. Furthermore, the BLOC-OP, being an organisational technology, is not considered a medical device. Therefore, the CE marking, with its associated procedures, does not apply in this case [7].

Domain 5: Safety

The BLOC-OP implements several measures to guarantee the security of the system. Access to the local network, which is used for communication between devices, requires authentication using Wi-Fi Protected Access 2 (WP2) encryption. WP2 encryption is a security standard that uses advanced cryptographic algorithms to protect the Wi-Fi connection from unauthorised access and passive or active attacks. In addition, access to the local network is subject to MAC address constraints. The MAC address is a unique identifier assigned to each network device. The use of MAC address constraints makes it possible to limit network access to authorised devices only, preventing access by unauthorised or unrecognised devices.

The storage system adopted by the BLOC-OP is based on the non-relational database MongoDB. MongoDB is a NoSQL database that offers a flexible structure for organising data and provides advanced security features. This includes the ability to encrypt data access, ensuring that only authorised users can access sensitive information. To ensure data reliability and availability, the system uses Redundant Array of Independent Disks (RAID) media. RAID is a technology that combines multiple hard disks into a single logical drive, providing redundancy and reducing the risk of data loss. RAID media allow data to be replicated and distributed across multiple disks, ensuring continuity of operations even in the event of hardware failure or read/write errors. The authors also point out that the components of the medical record traceability system are not considered medical devices according to the Italian Legislative Decree 46/1997 [26]. Furthermore, these components will not interact with the computer systems already in use at the hospital. Compatibility with existing systems has been confirmed by the Clinical Engineering Departmental Structure.

Domain 6: Clinical aspects

While it can be inferred that there will be important positive effects for patients secondary to improved intervention planning, as it is an organisational technology, it is more difficult to determine the exact clinical effects (e.g. in terms of effects on mortality, morbidity, QoL, etc.). The authors hope to be able to measure these factors at a later stage, after the actual implementation of the scheduling system with AI.

Domain 7: Economic aspects

The authors claim, as one of the secondary objectives of the study, that an analysis of the impact of the system in terms of the economy of the operating block will be carried out. It is hypothesised that a more efficient organisational system that allows an increase in the number of operations performed can also have positive effects on hospital economics. One of the secondary objectives is an analysis of the impact of this organisational system on the economy of the operating block, considering both the negative impact of the costs of developing and implementing the system, and the positive impact of the savings secondary to the efficiency of the scheduling.

Domain 8: Organisational aspects

After the training of the AI algorithm, the BLOC-OP workflow will be able to process the variables necessary to draw up an appropriate operating block schedule, starting with the basic logic concerning the presence of doctors on shifts and the availability of operating theatres. Once the waiting list of patients has been pre-arranged, the system will use this information to define the planning of interventions, also ensuring a correct order according to the priority codes assigned to individual patients. The ultimate goal is to create a technology that, taking into account all the relevant information for this decision-making process, styles the most effective planning possible, in a completely autonomous manner. In addition, multidisciplinary work is planned with a team of management engineers to arrive at improvement proposals in a strictly organisational sense.

Domain 9: Patient aspects

In the context of the BLOC-OP, the risk/benefit assessment ensures that patients are not exposed to additional risks compared to standard treatment. The technology, at this stage, is simply based on monitoring the timing of surgical operations, without changing the way patients are treated or their time on the waiting list. There is therefore neither a direct benefit nor an additional risk at the technology development stage. The benefits could emerge in the event that actual deployment reduces waiting times and offers better access to care.

In addition, a collection of reports from patients involved in the study is planned to collect any feedback from them. There is currently a client interface, hosted by the central server, that offers several functionalities. Patient enrolment is done by a member of the medical staff when the patient enters the operating block to undergo surgery. The enrolment is performed using a tablet device that communicates with the web application and by providing the necessary information. The system also provides a graphical interface able to monitor in real time the various movements of patients.

Process factors for a MAS-AI assessment

The so-called ‘process factors’ of MAS-AI assessment were submitted to the project team of the BLOC-OP in order to obtain a self-assessment of certain aspects of project development (Table 2). We report directly the answers received.

Table 2 The assessment of the process factors1Summary table of the assessment

The MAS-AI model suggests that at the end of a full assessment a summary table should be produced (Table 3).

Table 3 Structured analysis of the BLOC-OP project according to the MAS-AI model criteria