SkillsBench
Compare how skills perform across complete agent configurations, including the model, harness, tools, environment, artifact, and trace.
Show Us Your Agent Skills / EP 07 / guest dossier
HAN-CHUNG LEE
SKILLSBENCH
TRAJECTORIES
OBSIDIAN
NIGHTLY CODEX
HAN-CHUNG LEE
Han evaluates the complete agent: the model, the harness that gives it hands and feet, the skill, the tools, the controlled environment, the resulting artifact, and every action in between. SkillsBench makes those combinations comparable and exposes failures that a plausible final answer can hide. Then Han applies the same discipline to his own memory. He clips only articles he has read into Obsidian and lets a nightly Codex task connect entities and concepts across English, simplified Chinese, and traditional Chinese without rewriting his prose. Han is Director of Machine Learning at Moody's and a contributor to SkillsBench.
MODEL + HARNESSthe unit SkillsBench evaluates as an agent
FULL TRACEactions, tool calls, observations, and environment changes
CHROME?an unexpected tool invocation found in the trajectory
100 / 1,000sandboxed tasks in a scaled evaluation run
NIGHTLYthe second brain's sleep time
EN / 简 / 繁aliases across English and Chinese sources
EP 07 · HAN-CHUNG LEE · Claude Code as a control plane, SkillsBench, trajectory inspection, verifiable results, and a nightly Obsidian knowledge graph
"We define agents as having a harness plus a model, because the model needs hands and feet to connect it to different tools."
A benchmark score belongs to a complete configuration. The same skill can produce a different result when the model or harness changes. 00:47:40
THE WHOLE AGENT GETS EVALUATED
the benchmarking-agent-skills workflow, demonstrated through SkillsBench
SkillsBench does not assign a skill one universal score. Its leaderboard compares model-and-harness combinations with and without the skill, because the harness determines how the model receives instructions, calls tools, and interacts with the environment.
Han's live task asks an agent to turn a CSV into a quantitative sales-report workbook. The run happens in a controlled environment. A short Python verifier checks the workbook's structure, regional values, and formulas, including whether totals remain live spreadsheet formulas instead of hardcoded numbers.
The final artifact is only one part of the evidence. SkillsBench preserves the trajectory so an evaluator can see prompts, reasoning, actions, tool calls, observations, errors, and environment changes. That is how Han found a prior run invoking Chrome unexpectedly even though Chrome had nothing to do with the requested workbook.
"Nowadays the evaluation surface just grows so big in the agent world, because no longer do you only have the output, but you have the traces as how the agent actually completed its task."
A correct file can sit at the end of a wasteful, unsafe, or out-of-bounds trajectory. The path is part of the result. 00:53:07
TASK, SANDBOX, VERIFY, INSPECT
a benchmark that explains failures instead of stopping at a score
Define one task and its environmentTurn the prompt into a self-contained task with inputs, tools, dependencies, and a controlled workspace. 00:50:38
Choose the complete configurationRecord the skill, model, harness, and tool access as the agent under test. 00:48:22
Run inside a sandboxIsolate the task so the agent can act without contaminating other runs or touching unrelated systems. 00:50:38
Grade the artifactUse a short deterministic verifier when possible and an LLM judge when the criterion depends on taste. 00:55:04
Inspect the trajectoryRead the actions, tool calls, observations, and environment changes for reward hacking or unexpected behavior. 00:56:48
Scale and compareRun hundreds or thousands of tasks across sandboxes, then compare accuracy, behavior, reasoning, and token spend. 01:03:53
"We have to make sure the agent doesn't go out of bound."
A passing test can conceal a deleted test file. A trace can reveal the agent changing the yardstick instead of satisfying it. 00:57:07
THE TRACE TURNS FAILURE INTO A DIAGNOSIS
skill instruction, agent behavior, artifact, and verifier result line up in one record
The live workbook fails because its total is the hardcoded value 4050 instead of a live SUM formula. The trajectory explains why: the spreadsheet skill itself instructed the agent to calculate totals and write the values into the cells. The model followed the bad instruction and still reported success.
That evidence separates several possible repairs. A team can fix the skill, change the model-and-harness combination, constrain the available tools, strengthen the verifier, or add a trajectory check. It can also measure two successful agents differently by penalizing excessive reasoning or token spend.
Han connects this to loop engineering later in the episode. A verifier creates the signal that lets an agent keep working, but its reward must describe the desired outcome well enough that deleting the test, bypassing the check, or changing the environment does not count as success.
"It's a nightly job, sort of. I guess this is my second brain's sleep time when it does the agglomeration and tagging of the subject."
Once Han has done the reading, a scheduled agent handles the repetitive linking and index maintenance inside his vault. 01:09:13
THE VAULT WORKS WHILE HE SLEEPS
the nightly-knowledge-graph workflow for a maintained, multilingual LLM wiki
Han starts with a strict ingestion rule: only articles he has read enter the Obsidian vault. The corpus is his augmented memory, not a warehouse of pages that an automation collected while he looked away.
Each night, a scheduled Codex task sifts through the notes, extracts entity and concept names, refreshes a wiki index, and adds evidenced aliases to canonical pages. A transformer reference can then retrieve sources in English, simplified Chinese, and traditional Chinese, including material clipped from Weibo or Zhihu.
The graph makes tip-of-the-tongue retrieval possible when Han remembers an idea but has lost the keyword. The task also preserves human-authored prose and reports ambiguous maintenance work instead of silently rewriting it. The agent organizes the corpus; Han retains the reading and comprehension.
"Only the articles I read, I will clip them, store them in Obsidian. Again, I don't want to clip something I don't remember."
The scheduled job begins after the human has selected, read, and understood the source. 01:08:17
READ, CAPTURE, CONNECT, REVIEW
a principled scheduled workflow for keeping an LLM wiki useful
Read the sourceFinish the article before it earns a place in the personal corpus. 01:08:17
Clip it into ObsidianStore the source in portable Markdown inside a linkable vault. 01:08:09
Schedule the maintenance passPoint a recurring Codex task at the notes folder and run it during the vault's sleep time. 01:09:43
Extract entities and conceptsTag names, link concepts, and refresh the wiki index across the corpus. 01:08:51
Resolve multilingual aliasesConnect English and Chinese references to canonical pages without erasing the original source language. 01:11:07
Inspect the graph and reportUse clusters for retrieval, preserve human prose, and surface ambiguous changes for review. 01:14:45
"We should be learning the same way, just so that we don't completely outsource our understanding, our comprehension to the models."
The corpus remains valuable because Han read it. The model connects that understanding without pretending to replace it. 01:14:45
HAN'S FIELD KIT
the benchmark, two reusable workflows, and the verifier-centered loop they support
benchmarking-agent-skills
Define sandboxed tasks, grade their artifacts, inspect trajectories, and compare the behavior of model-and-harness combinations.
nightly-knowledge-graph
Maintain a multilingual Obsidian wiki with a scheduled agent while keeping reading, source selection, and prose with the human.
verifiable-agent-loops
Define independent completion evidence, constrain the worker, inspect the trajectory, and repair rewards the agent learns to game.
CLI control plane
Han uses Claude Code to work through AWS and Jira from one interface, with minimum-permission credentials for consequential systems.
Fork before the deep end
Branch a research or writing session before a long detour makes the accumulated context difficult to redirect or recover.
"At the end, the key is you have to have a verifier that you can verify the output is correct."
Better models need less procedural instruction, but a trustworthy loop still needs evidence that the work reached the intended state. 01:37:10
HAN'S STACK
agent evaluation, scheduled knowledge maintenance, and practical control planes