The Eticas AI Risk Taxonomy: Operationalizing AI Evaluations with Measurable Proof
Every organization deploying AI today can point to a page of principles: fairness, transparency, privacy, accountability. What almost none of them can point to is proof. How do you know your AI is actually fair, or actually private, in production — not just on paper? That's the gap our team set out to close, and it's the subject of a paper we've just published: The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits. It documents the taxonomy behind our AI system evaluations, and — more importantly — the method we use to turn a named risk into a measured, graded result on a real system.
NOTE: Further updates are captured on our github repository, and some details from the paper may soon be outdated/changed.
The problem: 74 AI risk taxonomies, and almost none of them are testable
AI risk taxonomies aren't in short supply. Researchers count at least 74 of them in circulation today, each one cataloguing risks like bias, privacy, or reliability. That's useful for building a shared vocabulary. It's not useful for an audit team standing in front of a real AI system asking: what exactly do we test, what do we measure, and how bad is the result?
That's the part almost every taxonomy skips. Naming a risk is the easy half. Turning it into a test you can run, a number you can trust, and a grade that holds up to scrutiny — that's the hard half, and it's the half our paper is about.
What we built: an AI audit methodology from principle to proof
We describe a four-layer methodology that we've developed and run over more than ten years of AI evaluations, across hundreds systems and twelve sectors, from healthcare to law enforcement to financial services. In short, it works like this:
A risk (say, "the system leaks personal data") is broken down into the concrete mechanisms through which it can actually happen — for personal data, that might mean it gets disclosed in a conversation, or memorized during training and later repeated.
Each mechanism is tested through a defined probe — a specific procedure designed to try to trigger the failure.
Each probe produces a metric: a number, not an impression.
Each metric maps onto a severity band (0 to 5), calibrated for that specific mechanism.
Severities roll up into a grade, from A to E, that tells you — and your regulator, and your board — how serious the finding is, and whether it's an isolated blip or a systemic pattern.
Every step is defined in advance, so two different auditors running the same test should land on the same answer.
Overview of the Eticas AI Risk Taxonomy Framework. The framework organizes AI risks through a hierarchical taxonomy of categories, sub-groups, and granular risk concepts, enriched with semantic attributes describing definitions, mechanisms, lifecycle stages, relationships, and mappings to external risk frameworks.
Seeing it work: a real result, not a hypothetical
The paper doesn't just describe this method — it runs it, in full, on a real model. Using PII leakage as the test case, the team measured how often GPT-4-0314 revealed personal information under a public benchmark (DecodingTrust), at three levels of pressure:
With no coaxing at all: 0% disclosure.
After being shown one example of the model leaking data: 51% disclosure.
After three examples: 84% disclosure.
That's not a small variation, but a model that looks fully protective on the easy test and fails the majority of the time under mild pressure. Averaged together, that swing would disappear into a single, reassuring-looking statistic. Measured properly, it earns the model's privacy protections the most severe grade the framework has: an E, flagged as a systemic — not isolated — failure.
This is the point we're making with the whole paper: a principle like "we protect personal data" can be true and false at the same time, depending entirely on how hard you look. Evaluation is what tells you which one you're actually getting.
The bigger picture: an open, mapped, and growing AI risk taxonomy
Around that worked example sits the full Eticas AI Risk Taxonomy v2.0.0: 10 top-level categories (including Bias and Fairness, Privacy and Confidentiality, Governance, Security and Misuse, and Agentic AI) broken down into 20 sub-groups and 76 subcategories, each one scoped to be testable through a single, coherent method.
We also mapped it against 18 external frameworks that organizations already have to answer to, including the EU AI Act, ISO/IEC 42001, the NIST AI frameworks, and OWASP's guidance for agentic systems. One finding from that exercise is worth calling out on its own: every major binding AI regulation was finalized before agentic AI — systems that plan, use tools, and act with limited human involvement — became a mainstream deployment pattern. The regulatory floor simply hasn't caught up yet, which is exactly why we treat Agentic AI as its own first-class category rather than an afterthought.
The category structure of the taxonomy is published under an open license (CC BY 4.0), with stable, machine-readable identifiers, at taxonomy.eticas.ai, so that other auditors, regulators, and researchers can build on it, map their own frameworks to it, and hold audits to a comparable standard — not just ours. This is a work in progress!
Why this matters
Plenty of organizations can write down what responsible AI should look like. Far fewer can show you the test, the number, and the grade behind that claim. That distance — between a stated principle and a measured result — is exactly where AI accountability tends to quietly fail.
In short: we took one risk (personal data leaking from an AI system), followed it end to end through a real test, and showed the exact point where a "safe" model stops being safe. That same method scales across 76 risk categories and maps to 18 global frameworks, and it's published openly so anyone auditing AI can hold their own work to the same standard.
This paper is our attempt to close that distance in public: not just telling people what to look for in an AI system, but showing, with real numbers on a real model, how to find it. We publish it because the field needs shared, open infrastructure for this kind of work, and because organizations navigating AI accountability deserve more than reassurance, They deserve evidence.
Read the full paper for the complete methodology, the taxonomy's category structure, and the framework mappings: The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits.