How to evaluate an AI expert: start with the work
A practical framework for evaluating AI expertise through artifacts, context, and clear limitations.
Ask for one relevant artifact
Start with a paper, repository, documented deployment, or detailed explanation that relates to your problem. A broad biography can tell you where to look; the artifact tells you what to examine. Ask what the person contributed, who else worked on it, and what is available for you to inspect.
Read the conditions alongside the result
A reported result only makes sense alongside its task, data, model version, and evaluation method. A polished demo may be useful evidence of a prototype. It is not, by itself, evidence of reliable production performance. Ask what failed and how often a person had to intervene.
Choose the right kind of expertise
A researcher may be the right partner for designing an evaluation. A Forward Deployed AI Engineer may help integrate an application with a customer’s systems. An educator may help a team understand and adopt it. These contributions complement each other; a universal popularity score hides the distinctions.
Use a bounded trial
Agree on a small task, a budget, the data that can be accessed, and a written definition of success. Decide what the system must never do without approval. Compare the outcome to your current process and include the cost of review and maintenance.
Check when the evidence was reviewed
Tools and affiliations change. Read the source date and the directory review date separately. A source check means the linked material was inspected; it does not certify every claim or imply an endorsement. Use the shortlist to compare evidence before starting a conversation.
Explore Proof of Work, browse AI experts, or read our editorial methodology.
For hands-on agent operations, visit AI Agent Insights.
Meet the people.
Then build something useful.
Explore practical agent operations, open source AI Employees, and a community of builders at Agent Ops Club.