Lessons Learned in Cognitive Search (EVA): Extracting Key Lessons from Independent Evaluations Using AI
Start of implementation: 2019
Technology type: Artificial intelligence
Technology service provider: ERNI
ITD’s Partner Department: Independent Evaluation Department (IED)
In line with ADB’s Operational Approaches:
• Promoting digital development and innovative technologies
• Delivering integrated solutions
• Adding value and promoting quality infrastructure
The “One ADB” approach explicitly states that ADB aims to bring together knowledge and expertise to effectively implement Strategy 2030.Thus, it was essential for the bank to have a system in place that allowed users to find the right information quickly and easily to support decision-making. This was particularly important considering that Developing Member Countries (DMCs) are now experiencing increasingly complex issues, including those brought by climate change. Identifying common themes and lessons may be useful to policymakers who face similar issues.
The Independent Evaluation Department (IED) is responsible for reviewing and extracting key lessons from ADB documents to feed into sector, corporate, and thematic evaluations and support the design of new operations. In the past, it had no platform in place that automatically generated and curated insights to provide users with information to guide them in decision-making. Instead, IED staff manually reviewed these documents to extract lessons. This was a challenging task considering that ADB’s institutional knowledge is spread across over 20,000 documents, including Project Completion Reports (PCRs) and Project Validation Reports (PVRs), with more added yearly. It also has research products that provide policy advice and different perspectives across Asia and the Pacific. Typically, staff needed half a day to review and extract lessons from one document.
A keyword-based enterprise search function was not sufficient for a platform intended to extract lessons from key ADB documents, especially since this traditional search tool heavily relied on the user knowing what keywords to use to get correct results. What was needed was a new intuitive system that could cull valuable insights quickly and easily from documents. Thus, ITD and IED collaborated to develop EVA.
A major task that had to be undertaken was to “teach” EVA to identify what are and are not lessons. ADB adopted the definition of “lesson” used by the International Labour Organization: “A lesson learned is an observation from project or programme experience which can be translated into relevant, beneficial knowledge by establishing clear causal factors and effects. It focuses on a specific design, activity, process, or decision and may provide either positive or negative insights on operational effectiveness and efficiency, impact on the achievement of outcomes, or influence on sustainability. The lesson should indicate, where possible, how it contributes to 1) reducing or eliminating deficiencies; or 2) building successful and sustainable practice and performance.”
Annotations also needed to be made to improve the accuracy of EVA. PCRs were initially loaded to the model, followed by PVRs, Technical Assistance Completion Reports, and Technical Assistance Completion Report Validation. IED staff were tasked with annotating and validating reports to provide initial inputs to the model as part of the teaching process.
Example of a lesson extracted by EVA.
The beta version of EVA.
The minimum viable product (MVP) was officially launched and identified for scaling in 2020. EVA identifies, generates, and ranks lessons according to importance and relevance based on what users searched for. The tool extracts lessons from thousands of ADB documents quickly, allowing users, particularly IED staff, to save time.
Users, meanwhile, can filter results by sector, country, theme, and year. They can also correct wrongly tagged lessons to help train the model to recognize what are and are not lessons.
The full version of EVA, once rolled out, is expected to offer numerous benefits, including making evaluation knowledge more accessible and used, increasing efficiency in reviewing and evaluating ADB documentation, and making information available on demand to support decision-making.