Systematic review and analysis processes have become a cornerstone in enhancing the performance of national institutions, particularly within the military domain. Operational effectiveness on the battlefield alone is no longer sufficient to ensure superiority and professional excellence. Instead, the ability to extract lessons learned and embrace continuous learning from real-world experience has emerged as a decisive factor in sustaining success.
In this context, operational and performance assessment processes have gained increasing importance within NATO, especially with the rise of Multi-Domain Operations (MDO). This evolving concept has necessitated a re-evaluation of traditional analytical approaches to align with a complex, interconnected, and rapidly changing operational environment.
At the forefront of this transformation is the Joint Analysis and Lessons Learned Centre (JALLC), operating under Allied Command Transformation. The centre plays a critical role in converting operational experience into structured, actionable knowledge. It provides evidence-based recommendations aimed at improving readiness, planning, and capability development across the Alliance and its member states. One of its most notable publications, the JALLC Analysis Handbook (2024), offers a comprehensive framework that bridges theoretical concepts with practical application, covering all stages of scientific analysis—from problem definition and methodological design to reporting and recommendation formulation.
Beyond serving as a procedural guide, the handbook represents a fully integrated institutional methodology. It proposes measurable, reviewable processes applicable not only within military organisations but also across institutions requiring structured data analysis and performance improvement systems. In doing so, it transcends the role of a technical manual to establish a broader intellectual framework that embeds scientific rigour into military assessment and analysis.
Analysing NATO’s Operational Assessment System
The handbook presents a practical case study illustrating how the Alliance addressed the institutional challenge of modernising its Operational Assessment System (OPSA) to meet the demands of large-scale, multi-domain operations. Central to this effort is the formulation of key analytical questions, which guide research focus, define scope, and establish priorities. These questions transform broad challenges into structured analytical frameworks, enabling the collection of relevant data, the identification of causal relationships, and the development of accurate, actionable conclusions.
In this case, the central question was defined as: how can NATO’s operational assessment system be enhanced to become more effective and responsive to large-scale, multi-domain operations, while incorporating organisational, procedural, and analytical improvements that ensure its future relevance?
Recognising that the quality of analytical outputs depends on the robustness of the methodology, the JALLC team adopted a comprehensive approach combining both quantitative and qualitative analysis. Data was collected from multiple command levels, operational documents, performance reports, and interviews with commanders and subject-matter experts. These inputs were then integrated into a unified analytical framework to avoid fragmentation and ensure consistency.
This approach enabled the development of a coherent and holistic understanding of causal relationships between activities, decisions, and outcomes. It also highlighted the interdependence between organisational structures and operational processes within the OPSA framework, forming the basis for actionable recommendations.
Methodology and Analytical Tools
The analysis began with defining the problem framework, identifying objectives, and contextualising the operational environment. OPSA was not treated as a standalone technical function but as a socio-technical system in which data, institutional culture, and decision-making processes interact dynamically.
A comprehensive data collection plan was implemented, incorporating structured interviews, document reviews, analysis of previous assessment reports, and field observations through site visits. To enhance pattern recognition across different levels, the team avoided segregating data by source, ensuring logical integration and preserving the interconnected nature of the information.
During the analytical phase, a suite of complementary tools was employed. Among the most prominent was the Ishikawa (Fishbone) Diagram, used to structure causal analysis by categorising influencing factors into human, organisational, technical, and temporal dimensions. This method enabled the identification of root causes rather than merely addressing surface-level symptoms.
Additionally, the Bow-Tie model was utilised to map the causes of system failures and their consequences, while identifying control measures and preventive mechanisms. Cause-and-effect analysis further supported the examination of relationships between data collection processes, measurement standards, and decision-making behaviour.
To integrate findings across different levels—tactical, operational, and strategic—the team employed Analytical Correlation Matrices. These tools facilitated a structured comparison of variables and outcomes, allowing for a balanced evaluation of performance across diverse operational environments.
Following the initial analysis, the findings underwent peer reviews and validity testing with field experts. This step ensured the mitigation of individual and organisational biases, strengthened the credibility of results, and confirmed the practical applicability of the conclusions.
The final output consisted of a set of actionable recommendations, including structural and procedural adjustments, the development of digital measurement tools, the standardisation of data collection protocols, and the enhancement of analytical training for personnel involved in operational assessment.
Key Findings and Lessons Learned
The study revealed that the effectiveness of an operational assessment system is not determined solely by technological capabilities. Rather, it depends fundamentally on the presence of a mature institutional analytical culture that views assessment as a tool for learning and improvement, rather than merely a mechanism for accountability.
One of the primary challenges identified was the fragmentation of information and the inconsistency of measurement methodologies across different entities and command levels. This often resulted in a fragmented picture of performance, where data failed to provide a coherent and accurate representation of reality. The study addressed this issue by advocating for a unified measurement and reporting framework that ensures comparability and links tactical indicators to clear operational and strategic objectives.
The integration of quantitative and qualitative analysis was also highlighted as a critical success factor. While quantitative data reveals trends and identifies strengths and weaknesses, qualitative analysis provides context, explaining decision-making processes and uncovering root causes.
Furthermore, the study emphasised the importance of focusing on causal relationships rather than merely describing observable symptoms. This approach enables the development of targeted, implementable recommendations that address underlying issues rather than superficial manifestations.
Among the practical lessons identified were the need to establish unified platforms for managing analytical data, integrate collaborative validation mechanisms within the analytical cycle, and continuously update tools and standards to keep pace with evolving operational environments and technological advancements.
Conclusion
The study demonstrates how systematic analysis can be transformed into a powerful instrument for institutional change. When review processes are conducted as structured learning activities—grounded in reliable data, supported by causal analysis tools, and validated through rigorous testing—they become strategic enablers that enhance readiness and improve decision-making efficiency.
The work of JALLC highlights that scientific analysis, when embedded within an institutional culture open to critique and knowledge exchange, produces realistic and actionable recommendations. More importantly, it fosters a knowledge-driven environment based on evidence, transparency, and continuous improvement.
The lessons derived from this experience extend beyond NATO, offering a valuable model for defence institutions seeking to build robust analytical capabilities and maintain a competitive edge in an increasingly complex and rapidly evolving security landscape.●
By: Major General (Ret.) Khaled Ali Al-Sumaiti
