Auditing AI Enabled Career Advisory Platform
As AI tools increasingly shape how young people explore their futures, independent evaluation is essential. In December 2025, Eticas.ai, an AI auditing organization specializing in impact assessments of AI systems, conducted an independent post-deployment audit of the Career Scoops AI-enabled career exploration and readiness platform* focusing on real-world performance for K–12 students in the United States. The audit was part of the Gates Foundation’s sponsored research program and focused on the end users in real educational settings. Eticas’ audit was holistic: it measured and analyzed Career Scoops’ production data, live outputs, and system documentation alongside Eticas’ four risk categories.
About Career Scoops
Career Scoops is an AI-powered career readiness and exploration platform that supports students (ages 13+), schools, and youth-serving organizations in navigating equitable, student-centered career discovery, reflection, and planning. The platform is designed to improve students’ outcomes and delivers this value through its research-based RISE framework:
Reflection & Self-Discovery (R): helps students surface their interests and strengths through career assessments to explore aligned career pathways.
Individual Reports (I): translate student insights into clear, actionable next steps connected to learning and development opportunities.
Strategic Insights (S): provide aggregate visibility reports into student career readiness, uncertainty, and pathway alignment to support strategic planning.
Exposure & Ongoing Guidance (E): offers personalized guidance through AI companions, helping extend access to career exploration and postsecondary planning.
About the Audit
The audit’s primary objective was: evaluate whether the AI system(s) themselves operate in line with Career Scoops’ mission. Do they provide students equitable, user-friendly career guidance? Are there any potential fairness, privacy, reliability, and security risks?
This audit follows a post-deployment impact assessment approach. Rather than evaluating theoretical design intentions alone, it assesses observable system behaviour using production data, live outputs, and system documentation.
The audit was conducted in two layers:
(1) A component-level review aligned with Career Scoops’ RISE framework; and
(2) A cross-cutting risk analysis across four domains: Bias & Fairness, Privacy & Confidentiality, Reliability & Manipulation, and Security & Misuse.
Quantitative testing was combined with qualitative review, including structured “LLM-as-a-judge” assessments to evaluate factuality, appropriateness, actionability, and potential risk signals.
Main Findings
Career Scoops’ AI platform demonstrates strong alignment with its mission and presents no evidence of systemic risk. Overall system quality is high, and identified risks are localized rather than systemic. The main findings corresponding to Career Scoops’ RISE framework are:
Reflection & Self-Discovery (R): students consistently shortlist careers that are statistically closer to their assessed interests than non-selected options, confirming the effectiveness of the recommendation logic.
Individual Reports (I): student reports score very highly on factuality, appropriateness, and actionability. Most hallucination flags relate to over-precise labor-market or salary references and occasional overstatements about traits being “essential,” rather than unsafe content.
Strategic Insights (S): aggregate school reports are internally coherent and well-grounded in student data but could be made more actionable.
Exposure & Ongoing Guidance (E): the AI companion (chatbot) performs reliably, with no evidence of systematic hallucinations, manipulation, or toxic language, and no meaningful differences in hallucination rates across gender, age, or location. There is no evidence of data leakage and out-of-scope chats, and adversarial attempts are typically refused or safely redirected.
Takeaway
Career Scoops demonstrates a safe, reliable, and student-appropriate deployment of AI for career exploration. Across the RISE framework, the system behaves largely as intended, with strong guardrails and limited, well-scoped risks. Areas for improvement are set out in the following recommendations:
Strengthen the actionability of the aggregate Strategic Insights reports by adding school-level ownership, timelines, and measurable outcomes.
Refine AI-generated language to reduce overstatement.
Improve traceability of targeted guidance for students with low certainty / no-interest profiles.
Formalize and enhance security practices through documentation, anomaly detection, and regular vulnerability testing.
Continue monitoring fairness, misuse, and edge-case behaviors throughout.
Career Scoops’ deployment reflects a maturing approach to AI integration in education technology. Continued monitoring, refinement, and structured governance will further strengthen its resilience as the platform scales.
*The audit assessed a specific version and configuration of Career Scoops as of December 2025.