News
AI · Data Reliability · LLM · Policy Analysis
SEI researchers conducted a pilot project using their AI Reader tool, which leverages large language models like ChatGPT, achieving an average of 85% accuracy and 69% to 90% consistency in systematic policy document analysis compared to human assessment, demonstrating its potential for large-scale research.
The SEI AI Reader, developed in 2024, was tested on four documents, undergoing approximately 10 iterative runs per document to calibrate prompts and refine questions. Initial broad questions yielded generic results, but iterative refinement, including specific query-question combinations and context variables, significantly improved output quality.
The tool successfully resisted hallucination of facts, a critical safeguard implemented through prompt design requiring page references and direct quotes. Despite some unavoidable inconsistencies, the findings suggest AI-generated data meets academic quality standards, offering a promising solution for analyzing vast volumes of climate policy evaluations and identifying patterns in successful policy implementation across diverse contexts.
The next step involves expanding the analysis to a larger dataset to validate scalability.
