{
  "schemaVersion": 1,
  "title": "IonBin controlled helper evaluation harness",
  "preparedOn": "2026-10-18",
  "contains": "Original authored harness source and README only; no upstream code or generated fixtures",
  "portableBundleExecuted": false,
  "validation": {
    "pythonAstParsed": 3,
    "javascriptSyntaxChecked": 1,
    "upstreamCodeExecutedDuringPackaging": false,
    "personalPathScan": "pass"
  },
  "files": [
    {
      "path": "README.md",
      "language": "markdown",
      "content": "# IonBin helper evaluation harness\n\nThis bundle contains original IonBin evaluation/reporting code only. It contains no upstream skill code, downloaded dependency, fixture, generated HTML, private account data or personal machine path. Relative `private/` and `fixtures/` names are disposable local folders the scripts create; they are not links to IonBin's vault.\n\n## What was actually run\n\nOn 2026-10-18 the original harness ran on Windows 11 build 26200, Python 3.12.10, Node 24.18.0, Chrome 155.0.8059.82, Git Bash 5.3.15, Poppler 26.07.0, pypdf 6.19.0, ReportLab 4.4.9, Pillow 12.3.0, pdf2image 1.17.0 and defusedxml 0.7.1. There were 29 expectations: 25 passed and 4 failed. Two failures confirmed one script-context injection in the pinned skill-creator viewer; the other observations were PDF diagnostic value echo and non-form PDF handling. Five Playwright checks used fake npx dispatch; one checked Bash startup. No real Playwright CLI was installed or executed.\n\nThe pinned Anthropic source revision was 34040c9c568585f6929bedeaad110ad08f079624; the historical OpenAI wrapper revision was 49f948faa9258a0c61caceaf225e179651397431. These observations do not describe current upstream HEAD.\n\n## Original versus portable source\n\nThe original measured command was `python -B run_evals.py --runtime-libs <existing-bundled-site-packages> --poppler-bin <poppler-binaries>`, followed by `node viewer_probe.mjs` and `python finalize_report.py`. The original Python harness resolved a standard Windows Git Bash location; the original browser harness resolved Playwright from an adjacent site project. Those dependency lookups are the only changes in the portable execution scripts. Each file records its original and portable SHA-256 hashes and changes.\n\n**The portable execution scripts were syntax-checked, not rerun.** Their source changes do not constitute fresh runtime evidence. Evaluation cases, fixtures generated by the code, expectations, minimal environments, browser request blocking and helper calls are unchanged. Cross-platform operation has not been verified; the measured environment was Windows.\n\n`finalize_report.py` is included unchanged for historical transparency. It contains report-only constants from the original run (including date, environment observations and summary prose). It was not designed as a generic reporter. For a new run, use the freshly generated `results.json` and `browser-results.json`; review/update those historical reporting constants before invoking the finalizer. Do not present its original numbers as new measurements.\n\n## Extract the bundle\n\nSave each `files` entry's UTF-8 `content` under its `path` in a new, dedicated folder. The only allowed paths are README.md, run_evals.py, fetch_sources.py, viewer_probe.mjs and finalize_report.py. Verify each SHA-256 against the bundle before executing. Do not automatically execute text merely because it appears in a JSON bundle.\n\n## Acquire dependencies and sources separately from execution\n\nUse a dedicated environment with no live credentials or account sessions. Dependency acquisition requires network access; evaluation execution should not.\n\n1. Use Python 3.12.10, Node 24.18.0, Git Bash, Chrome and Poppler matching the recorded versions where possible. Place Bash on PATH or set the task-specific `IONBIN_BASH` variable to its executable. Keep Poppler binaries in a separate known directory.\n2. In the dedicated folder run `npm install --save-exact playwright-core@1.64.1`. The harness uses installed Chrome via its channel; playwright-core does not install a browser here.\n3. Install the other Python packages in that environment: `python -m pip install --only-binary=:all: pypdf==6.19.0 reportlab==4.4.9 Pillow==12.3.0 pdf2image==1.17.0`.\n4. Install the one private local wheel dependency: `python -m pip install --target private/deps --no-deps --only-binary=:all: --disable-pip-version-check defusedxml==0.7.1`. This is an additional package, not a replacement XML parser stub.\n5. Review `fetch_sources.py`, the exact upstream files and their applicable licenses before acquisition/execution. If your use is permitted, `python fetch_sources.py` acquires only the explicit public commit-pinned list and writes hashes into source-manifest.json. The DOCX dependency is **scripts/office/helpers/__init__.py**, a package entrypoint; helpers.py is not the correct source path.\n6. Inspect every retrieved helper and verify the recorded hashes against the public evaluation evidence. Fetching successfully or matching a hash does not establish trust or licensing permission. Anthropic PDF/DOCX files have proprietary/source-available terms; this bundle does not redistribute them or grant rights to them.\n\nKeep private/, fixtures/, node_modules/ and local Python environments out of publication. No upstream source or generated upstream-derived HTML should be copied into public reports.\n\n## Run the controlled probes\n\nAfter source inspection and acquisition, disconnect external networking or use a suitably isolated test environment. The original run was supervised local execution, not an OS sandbox. Docker was unavailable and WSL was not installed. Application guards constrain the reviewed test cases but do not contain arbitrary hostile native code.\n\n1. Run `python -B run_evals.py --poppler-bin <poppler-binaries>`. When relying on an existing separate Python package directory, additionally pass `--runtime-libs <site-packages>`.\n2. Inspect results.json. The harness generates synthetic PDF/XML/ZIP/viewer inputs and keeps writes within its local fixture area. It uses child deadlines and excludes credential environment variables. Its Python socket guards do not constitute an OS-level network policy.\n3. Run `node viewer_probe.mjs`. It loads only generated synthetic HTML in fresh offline Chrome contexts, blocks all page requests, and checks a harmless in-memory marker. It neither reads confidential data nor sends it anywhere. Do not replace the benign marker fixture with real credentials or target a live service.\n4. Inspect browser-results.json. These raw reports contain the new run's actual counts and observations. The report finalizer is historical as explained above.\n\nThe viewer cleanup probe mocks subprocess/os.kill effects; it never terminates a real process. The Playwright wrapper dispatches to a fake npx; it never executes npm packages or live browser workflows. Valid PDF rasterization uses the real installed Poppler tools on a small synthetic file. DOCX checks exercise the real merge/extraction helpers on bounded synthetic XML/ZIP inputs, including an internal entity and parent-path case that remains within test fixtures even if rejection fails.\n\n## Interpretation\n\nCount failed expectations, not vulnerabilities. Preserve blocked or failed setup attempts as environment limitations rather than skill findings. One deterministic pass per case was recorded in the original run. Passing these probes does not establish whole-skill effectiveness, a safe installation, full compatibility, current-version security or containment of arbitrary malicious input.\n",
      "sha256": "6b60402163c742fcc415ad88a6c83c91198f788daeaf02cc00872e36df4bf9b1"
    },
    {
      "path": "run_evals.py",
      "language": "python",
      "content": "\"\"\"IonBin controlled helper probes. Upstream code remains in ignored private/.\n\nRun: python run_evals.py --runtime-libs <existing-bundled-site-packages>\nNo agent prompts, live accounts or real confidential data are used.\n\"\"\"\nimport argparse\nimport contextlib\nimport hashlib\nimport importlib.util\nimport io\nimport json\nimport os\nfrom pathlib import Path\nimport platform\nimport shutil\nimport subprocess\nimport sys\nimport time\nimport zipfile\n\nROOT = Path(__file__).resolve().parent\nPRIVATE = ROOT / \"private\"\nFIXTURES = ROOT / \"fixtures\"\nRESULTS = []\n\n\ndef record(case, skill, mode, expectation, met, observed, **extra):\n    RESULTS.append(dict(id=case, skill=skill, mode=mode, expectation=expectation,\n                        outcome=\"pass\" if met else \"finding\", observed=observed, **extra))\n\n\ndef load(name, path):\n    spec = importlib.util.spec_from_file_location(name, path)\n    module = importlib.util.module_from_spec(spec)\n    sys.modules[name] = module\n    spec.loader.exec_module(module)\n    return module\n\n\ndef safe_label(value):\n    text = str(value).replace(str(ROOT), \"<evaluation-root>\").replace(str(ROOT).replace('\\\\', '/'), \"<evaluation-root>\")\n    return text[:1200]\n\n\ndef setup_child(runtime):\n    if runtime:\n        sys.path.append(runtime)\n    sys.path.insert(0, str(PRIVATE / \"deps\"))\n    sys.dont_write_bytecode = True\n    import socket\n    def no_network(*args, **kwargs):\n        raise PermissionError(\"Network disabled by evaluation harness\")\n    socket.create_connection = no_network\n    socket.socket.connect = no_network\n    socket.socket.connect_ex = no_network\n    socket.socket.bind = no_network\n    socket.getaddrinfo = no_network\n    # This is an application guard for known reviewed Python helpers, not an OS sandbox.\n    def guard(event, args):\n        if event in (\"socket.connect\", \"socket.bind\", \"socket.getaddrinfo\"):\n            raise PermissionError(\"Network disabled by evaluation audit hook\")\n        if event == \"open\" and isinstance(args[0], (str, bytes, os.PathLike)):\n            path = Path(os.fsdecode(args[0])).resolve()\n            mode = str(args[1] or '')\n            flags = args[2] if len(args) > 2 else 0\n            if any(c in mode for c in 'wax+') or (isinstance(flags, int) and flags & (os.O_WRONLY | os.O_RDWR | os.O_CREAT | os.O_TRUNC)):\n                if not path.is_relative_to(FIXTURES):\n                    raise PermissionError(\"Write outside synthetic fixtures blocked\")\n    sys.addaudithook(guard)\n\n\ndef pdf_suite():\n    sys.path.insert(0, str(PRIVATE / \"pdf\"))\n    filler = load(\"ionbin_pdf_filler\", PRIVATE / \"pdf/fill_fillable_fields.py\")\n    info = sys.modules['extract_form_field_info']\n    from pypdf import PdfReader\n    from reportlab.pdfgen.canvas import Canvas\n    folder = FIXTURES / \"pdf\"\n    folder.mkdir(parents=True, exist_ok=True)\n    specimen = folder / 'form.pdf'\n    canvas = Canvas(str(specimen), pagesize=(300, 200))\n    canvas.drawString(30, 175, 'Synthetic IonBin evaluation form')\n    canvas.acroForm.textfield(name='full_name', x=30, y=120, width=200, height=20)\n    canvas.acroForm.checkbox(name='consent', x=30, y=70, checked=False)\n    canvas.showPage()\n    canvas.save()\n    fields = info.get_field_info(PdfReader(specimen))\n    record('pdf-discover', 'anthropic-pdf', 'actual-helper', 'Discover the two synthetic fields with page 1 locations',\n           len(fields) == 2 and all(x['page'] == 1 for x in fields), {'fieldCount': len(fields), 'types': [x['type'] for x in fields]})\n    filler.monkeypatch_pydpf_method()\n    blank_hash = hashlib.sha256(specimen.read_bytes()).hexdigest()\n    cases = [\n        ('nominal', [{'field_id':'full_name','page':1,'value':'IonBin Sample'}, {'field_id':'consent','page':1,'value':'/Yes'}]),\n        ('invalid-checkbox', [{'field_id':'consent','page':1,'value':'SYNTHETIC_VALUE_MARKER'}]),\n        ('wrong-page', [{'field_id':'full_name','page':2,'value':'sample'}]),\n        ('unknown-field', [{'field_id':'missing','page':1,'value':'sample'}]),\n    ]\n    for name, payload in cases:\n        values = folder / f'{name}.json'\n        output = folder / f'{name}.pdf'\n        values.write_text(json.dumps(payload), encoding='utf-8')\n        captured = io.StringIO()\n        exit_code = 0\n        try:\n            with contextlib.redirect_stdout(captured):\n                filler.fill_pdf_fields(str(specimen), str(values), str(output))\n        except SystemExit as exc:\n            exit_code = exc.code\n        if name == 'nominal':\n            actual = PdfReader(output).get_fields()\n            observed = {'full_name':str(actual['full_name'].get('/V')), 'consent':str(actual['consent'].get('/V')),\n                        'sourceUnchanged':hashlib.sha256(specimen.read_bytes()).hexdigest() == blank_hash}\n            record('pdf-fill', 'anthropic-pdf', 'actual-helper', 'Fill text/checkbox and preserve source when output differs',\n                   observed['full_name'] == 'IonBin Sample' and observed['consent'] == '/Yes' and observed['sourceUnchanged'], observed)\n        else:\n            record('pdf-' + name, 'anthropic-pdf', 'actual-helper', 'Reject invalid field data without producing a PDF',\n                   exit_code == 1 and not output.exists(), {'exitCode':exit_code,'outputCreated':output.exists(),'diagnostic':captured.getvalue().strip()})\n            if name == 'invalid-checkbox':\n                echoed = 'SYNTHETIC_VALUE_MARKER' in captured.getvalue()\n                record('pdf-diagnostic-value', 'anthropic-pdf', 'actual-helper', 'Avoid echoing submitted field values in diagnostics',\n                       not echoed, {'syntheticValueEchoed':echoed}, consequence='Invalid choice values appear in diagnostics; logs need document-equivalent confidentiality.')\n    plain = folder / 'no-form.pdf'\n    canvas = Canvas(str(plain), pagesize=(200,200)); canvas.drawString(10,100,'No form'); canvas.showPage(); canvas.save()\n    try:\n        plain_fields = info.get_field_info(PdfReader(plain))\n        record('pdf-no-form','anthropic-pdf','actual-helper','Return an empty field list for a valid non-form PDF', plain_fields == [], {'fieldCount':len(plain_fields)})\n    except Exception as exc:\n        record('pdf-no-form','anthropic-pdf','actual-helper','Return an empty field list for a valid non-form PDF',False,{'exception':type(exc).__name__,'message':safe_label(exc)})\n    malformed = folder / 'malformed.pdf'; malformed.write_bytes(b'%PDF-1.4\\nSynthetic invalid file\\n')\n    try:\n        info.get_field_info(PdfReader(malformed))\n        record('pdf-malformed','anthropic-pdf','actual-helper','Reject malformed PDF',False,{'rejected':False})\n    except Exception as exc:\n        record('pdf-malformed','anthropic-pdf','actual-helper','Reject malformed PDF',True,{'exception':type(exc).__name__})\n    if os.environ.get('IONBIN_POPPLER_BIN'):\n        converter=load('ionbin_pdf_images',PRIVATE/'pdf/convert_pdf_to_images.py')\n        images=folder/'images';images.mkdir(exist_ok=True)\n        with contextlib.redirect_stdout(io.StringIO()):\n            converter.convert(str(specimen),str(images),max_dim=400)\n        from PIL import Image\n        with Image.open(images/'page_1.png') as image:\n            size=image.size\n        record('pdf-rasterize','anthropic-pdf','actual-helper','Rasterize one synthetic PDF page and constrain its longest dimension to 400 pixels',\n               max(size)==400,{'pages':len(list(images.glob('page_*.png'))),'pixels':list(size)})\n\n\ndef docx_suite():\n    sys.path.insert(0, str(PRIVATE / 'docx'))\n    merger = load('ionbin_docx_merge', PRIVATE / 'docx/merge_runs.py')\n    folder = FIXTURES / 'docx'; folder.mkdir(parents=True,exist_ok=True)\n    prefix = '<w:document xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\"><w:body><w:p>'\n    suffix = '</w:p></w:body></w:document>'\n    specimens = [\n        ('nominal', '<w:r><w:t xml:space=\"preserve\">Hello </w:t></w:r><w:r><w:t>world</w:t></w:r>',1,'Hello world'),\n        ('format-boundary','<w:r><w:rPr><w:b/></w:rPr><w:t>Hello</w:t></w:r><w:r><w:t>world</w:t></w:r>',0,None),\n        ('revision-boundary','<w:ins w:id=\"1\"><w:r><w:t>Hello</w:t></w:r></w:ins><w:ins w:id=\"2\"><w:r><w:t>world</w:t></w:r></w:ins>',0,None),\n        ('whitespace','<w:r><w:t>Hello </w:t></w:r><w:r><w:t> world</w:t></w:r>',1,'Helloworld'),\n    ]\n    for name, body, expected_merges, text in specimens:\n        target = folder / name / 'word'; target.mkdir(parents=True,exist_ok=True)\n        xml = target / 'document.xml'; xml.write_text(prefix + body + suffix, encoding='utf-8')\n        merges, message = merger.merge_runs(str(target.parent))\n        output = xml.read_text(encoding='utf-8')\n        record('docx-' + name,'anthropic-docx','actual-helper','Preserve text/format/revision semantics while merging only eligible runs',\n               merges == expected_merges and (text is None or text in output), {'merges':merges,'expectedMerges':expected_merges,'message':message,'expectedTextPresent':text is None or text in output})\n    for name, content, rejection in [\n        ('malformed','<w:document>','Error'),\n        ('entity','<!DOCTYPE w:document [<!ENTITY specimen \"SYNTHETIC_ENTITY\">]>' + prefix + '<w:r><w:t>&specimen;</w:t></w:r>' + suffix,'EntitiesForbidden'),\n    ]:\n        target=folder/name/'word';target.mkdir(parents=True,exist_ok=True)\n        xml=target/'document.xml';xml.write_text(content,encoding='utf-8')\n        count,message=merger.merge_runs(str(target.parent))\n        record('docx-'+name,'anthropic-docx','actual-helper','Reject malformed/entity XML and preserve original input',\n               rejection in message and xml.read_text(encoding='utf-8') == content, {'message':safe_label(message),'inputUnchanged':xml.read_text(encoding='utf-8') == content})\n    # This path remains entirely inside the synthetic-fixture root even if rejection fails.\n    payload=folder/'traversal.docx'\n    with zipfile.ZipFile(payload,'w') as archive:\n        archive.writestr('../synthetic-escape.txt','Synthetic traversal marker')\n        archive.writestr('word/document.xml',prefix+suffix)\n    dest=folder/'extract';dest.mkdir(exist_ok=True)\n    rejected=False; error=None\n    try:\n        with zipfile.ZipFile(payload) as archive:\n            merger.safe_extract(archive,dest)\n    except Exception as exc:\n        rejected=True;error=type(exc).__name__\n    escaped=(folder/'synthetic-escape.txt').exists()\n    record('docx-traversal','anthropic-docx','actual-helper','Reject parent-path ZIP entry before it escapes extraction directory',rejected and not escaped,{'rejected':rejected,'exception':error,'escapedFileCreated':escaped})\n\n\ndef viewer_suite():\n    from unittest.mock import patch\n    from types import SimpleNamespace\n    viewer=load('ionbin_skill_viewer',PRIVATE/'skill-creator/generate_review.py')\n    folder=FIXTURES/'viewer';folder.mkdir(parents=True,exist_ok=True)\n    run=folder/'workspace'/'eval-1'/'with_skill';(run/'outputs').mkdir(parents=True,exist_ok=True)\n    (run/'eval_metadata.json').write_text(json.dumps({'prompt':'Synthetic benign task','eval_id':1}),encoding='utf-8')\n    (run/'outputs'/'answer.txt').write_text('Synthetic evaluation output',encoding='utf-8')\n    (run/'outputs'/'metrics.json').write_text('{}',encoding='utf-8')\n    runs=viewer.find_runs(folder/'workspace')\n    record('viewer-discovery','anthropic-skill-creator','actual-helper','Discover one run and exclude metadata output',len(runs)==1 and len(runs[0]['outputs'])==1,{'runCount':len(runs),'outputCount':len(runs[0]['outputs'])})\n    html=viewer.generate_html(runs,'Synthetic skill')\n    (folder/'benign.html').write_text(html,encoding='utf-8')\n    record('viewer-static','anthropic-skill-creator','actual-helper','Generate static HTML containing the synthetic result', 'Synthetic evaluation output' in html and '/*__EMBEDDED_DATA__*/' not in html,{'htmlBytes':len(html.encode()),'externalScriptReference': 'https://cdn.sheetjs.com/' in html})\n    # Payload changes only an in-memory browser marker. No data is read or transmitted.\n    probe='</script><script>globalThis.__IONBIN_PROBE__=1</script><script>'\n    probe_runs=[dict(runs[0],prompt=probe)]\n    probe_html=viewer.generate_html(probe_runs,'Synthetic skill')\n    (folder/'script-probe.html').write_text(probe_html,encoding='utf-8')\n    record('viewer-script-boundary','anthropic-skill-creator','actual-helper','Escape closing-script text inside embedded JSON',probe not in probe_html,{'rawScriptBreakoutPresent':probe in probe_html},consequence='Untrusted prompt/output text can escape the JSON script context; browser marker confirmation is separate.')\n    calls=[]\n    with patch.object(viewer.subprocess,'run',return_value=SimpleNamespace(stdout='12345\\ninvalid\\n67890\\n')) as dispatch, patch.object(viewer.os,'kill',side_effect=lambda pid,sig:calls.append([pid,int(sig)])), patch.object(viewer.time,'sleep'):\n        viewer._kill_port(3117)\n    record('viewer-port-dispatch','anthropic-skill-creator','mocked-effects','Record exact cleanup intent without terminating any process',len(calls)==2,{'mockPids':calls,'lsofArguments':dispatch.call_args.args[0],'realProcessTermination':False})\n    with patch.object(viewer.subprocess,'run',side_effect=FileNotFoundError),contextlib.redirect_stderr(io.StringIO()) as err:\n        viewer._kill_port(3117)\n    record('viewer-no-lsof','anthropic-skill-creator','mocked-effects','Handle unavailable lsof without process effects','lsof not found' in err.getvalue(),{'diagnostic':err.getvalue().strip()})\n\n\ndef playwright_suite():\n    folder=FIXTURES/'playwright';folder.mkdir(parents=True,exist_ok=True)\n    bash_location = os.environ.get('IONBIN_BASH') or shutil.which('bash')\n    if not bash_location:\n        raise RuntimeError('Set IONBIN_BASH to an installed Git Bash executable')\n    bash=Path(bash_location)\n    fakebin=folder/'bin';fakebin.mkdir(exist_ok=True)\n    (fakebin/'npx').write_text('#!/bin/bash\\nprintf \"%s\\\\n\" \"$@\"\\n',encoding='utf-8',newline='\\n')\n    wrapper=PRIVATE/'playwright/playwright_cli.sh'\n    version=subprocess.run([str(bash),'--version'],capture_output=True,text=True,timeout=10)\n    record('playwright-bash-environment','openai-playwright','environment-check','Start installed Git Bash successfully',version.returncode==0,{'version':version.stdout.splitlines()[0] if version.stdout else 'unavailable'})\n    prefix=['--yes','--package','@playwright/cli','playwright-cli']\n    cases=[('nominal',['open','https://example.invalid/'],None,prefix+['open','https://example.invalid/']),\n           ('environment-session',['snapshot'],'fixture-session',prefix+['--session','fixture-session','snapshot']),\n           ('explicit-session',['--session=explicit','snapshot'],'ignored',prefix+['--session=explicit','snapshot']),\n           ('literal-argument',['fill','$(touch SHOULD_NOT_EXIST); space value'],None,prefix+['fill','$(touch SHOULD_NOT_EXIST); space value'])]\n    for name,args,session,expected in cases:\n        env=dict(os.environ);env['PATH']=str(fakebin).replace('\\\\','/');env.pop('PLAYWRIGHT_CLI_SESSION',None)\n        if session:env['PLAYWRIGHT_CLI_SESSION']=session\n        result=subprocess.run([str(bash),'--noprofile','--norc',str(wrapper),*args],cwd=folder,env=env,capture_output=True,text=True,timeout=10)\n        actual=result.stdout.splitlines()\n        record('playwright-'+name,'openai-playwright','mocked-dispatch','Pass expected CLI arguments literally to a fake npx; never install or launch CLI',result.returncode==0 and actual==expected,{'exitCode':result.returncode,'arguments':actual,'realNpxExecuted':False,'realBrowserExecuted':False})\n    env=dict(os.environ);env['PATH']=str(folder/'absent')\n    result=subprocess.run([str(bash),'--noprofile','--norc',str(wrapper),'snapshot'],cwd=folder,env=env,capture_output=True,text=True,timeout=10)\n    record('playwright-no-npx','openai-playwright','mocked-dispatch','Fail clearly when npx is absent',result.returncode==1 and 'npx is required' in result.stderr,{'exitCode':result.returncode,'diagnostic':result.stderr.strip()})\n\n\ndef main():\n    parser=argparse.ArgumentParser()\n    parser.add_argument('--runtime-libs',default='')\n    parser.add_argument('--poppler-bin',default='')\n    parser.add_argument('--suite',choices=['pdf','docx','viewer','playwright'])\n    args=parser.parse_args()\n    if args.suite:\n        setup_child(args.runtime_libs)\n        started=time.perf_counter()\n        try:\n            globals()[args.suite+'_suite']()\n        except Exception as exc:\n            record(args.suite+'-harness-error','harness','execution-error','Complete suite without setup failure',False,{'exception':type(exc).__name__,'message':safe_label(exc)})\n        print(json.dumps({'suite':args.suite,'elapsedMs':round((time.perf_counter()-started)*1000),'cases':RESULTS}))\n        return\n    FIXTURES.mkdir(exist_ok=True)\n    (FIXTURES/'temp').mkdir(exist_ok=True)\n    (FIXTURES/'home').mkdir(exist_ok=True)\n    manifest=json.loads((ROOT/'source-manifest.json').read_text(encoding='utf-8'))\n    for item in manifest:\n        if item['status']=='retrieved':\n            actual=hashlib.sha256((ROOT/item['localSource']).read_bytes()).hexdigest()\n            if actual!=item['sha256']:raise ValueError('Upstream source hash mismatch')\n    env={k:os.environ[k] for k in ['SystemRoot','WINDIR','COMSPEC'] if k in os.environ}\n    bash_location = os.environ.get('IONBIN_BASH') or shutil.which('bash')\n    if bash_location:\n        env['IONBIN_BASH'] = bash_location\n    env.update(PATH=str(Path(sys.executable).parent),TEMP=str(FIXTURES/'temp'),TMP=str(FIXTURES/'temp'),HOME=str(FIXTURES/'home'),USERPROFILE=str(FIXTURES/'home'),PYTHONIOENCODING='utf-8',PYTHONDONTWRITEBYTECODE='1')\n    if args.poppler_bin:\n        env['PATH'] += os.pathsep + args.poppler_bin\n        env['IONBIN_POPPLER_BIN'] = args.poppler_bin\n    suites=[]\n    for suite in ['pdf','docx','viewer','playwright']:\n        command=[sys.executable,'-B',str(Path(__file__).resolve()),'--suite',suite,'--runtime-libs',args.runtime_libs]\n        try:\n            completed=subprocess.run(command,cwd=ROOT,env=env,capture_output=True,text=True,encoding='utf-8',timeout=30)\n            data=json.loads(completed.stdout.strip().splitlines()[-1])\n            data['exitCode']=completed.returncode\n            data['diagnostics']=safe_label(completed.stderr)\n            suites.append(data)\n        except Exception as exc:\n            suites.append({'suite':suite,'error':type(exc).__name__,'cases':[]})\n    cases=[case for suite in suites for case in suite.get('cases',[])]\n    import importlib.metadata\n    if args.runtime_libs:sys.path.append(args.runtime_libs)\n    sys.path.insert(0,str(PRIVATE/'deps'))\n    versions={name:importlib.metadata.version(name) for name in ['pypdf','reportlab','Pillow','pdf2image','defusedxml']}\n    report={'schemaVersion':1,'reviewedOn':'2026-10-18','scope':'Controlled helper probes; not end-to-end skill effectiveness or security certification',\n            'environment':{'python':platform.python_version(),'os':platform.platform(),'gitBash':'available','docker':'not available','wsl':'not installed','packages':versions},\n            'controls':['Fresh minimal child environment; account/credential variables excluded','Synthetic fixtures only; Python writes guarded to fixtures','Python sockets disabled; browser probe separately blocks all network','30 second suite deadline; 10 second mocked Bash deadline','Known source hashes verified; upstream code not modified'],\n            'isolationLimits':['Supervised local processes, not an OS sandbox','No container, VM or memory/resource quota','Application guards do not establish containment of arbitrary hostile code','No real credential, remote account, or destructive-effect testing'],\n            'command':'python run_evals.py --runtime-libs <existing-bundled-site-packages> --poppler-bin <poppler-binaries>','suites':suites,\n            'summary':{'cases':len(cases),'passed':sum(c['outcome']=='pass' for c in cases),'findings':sum(c['outcome']=='finding' for c in cases)},\n            'upstreamSources':[{k:v for k,v in x.items() if k!='localSource'} for x in manifest]}\n    (ROOT/'results.json').write_text(json.dumps(report,indent=2),encoding='utf-8')\n    print(json.dumps(report['summary']))\n\n\nif __name__=='__main__':main()\n",
      "sha256": "75a1f64ab475ac3bec2f6bf8764add9c9f29e6422e514038b2f562bb89fa03ac",
      "originalExecutedSourceSha256": "7dac1f89e5f7cb0256ac8c0eaf82f6b47d8feb7c50460708de88cd4a40f5abb4",
      "adaptations": [
        "Resolve Git Bash through IONBIN_BASH/PATH and pass only that resolved location into the minimal child environment."
      ],
      "executionStatus": "Portable dependency lookup changed; syntax checked only, not rerun."
    },
    {
      "path": "fetch_sources.py",
      "language": "python",
      "content": "\"\"\"Download explicitly selected public upstream source for private evaluation only.\"\"\"\nimport hashlib\nimport json\nfrom pathlib import Path\nfrom urllib.request import Request, urlopen\n\nROOT = Path(__file__).resolve().parent\nANTHROPIC = \"34040c9c568585f6929bedeaad110ad08f079624\"\nOPENAI = \"49f948faa9258a0c61caceaf225e179651397431\"\nFILES = [\n    (\"anthropics/skills\", ANTHROPIC, \"skills/skill-creator/eval-viewer/generate_review.py\", \"skill-creator/generate_review.py\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/skill-creator/eval-viewer/viewer.html\", \"skill-creator/viewer.html\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/skill-creator/LICENSE.txt\", \"skill-creator/LICENSE.txt\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/docx/scripts/merge_runs.py\", \"docx/merge_runs.py\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/docx/scripts/office/helpers/__init__.py\", \"docx/office/helpers/__init__.py\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/docx/LICENSE.txt\", \"docx/LICENSE.txt\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/pdf/scripts/fill_fillable_fields.py\", \"pdf/fill_fillable_fields.py\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/pdf/scripts/extract_form_field_info.py\", \"pdf/extract_form_field_info.py\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/pdf/scripts/convert_pdf_to_images.py\", \"pdf/convert_pdf_to_images.py\"),\n    (\"anthropics/skills\", ANTHROPIC, \"skills/pdf/LICENSE.txt\", \"pdf/LICENSE.txt\"),\n    (\"openai/skills\", OPENAI, \"skills/.curated/playwright/scripts/playwright_cli.sh\", \"playwright/playwright_cli.sh\"),\n    (\"openai/skills\", OPENAI, \"skills/.curated/playwright/LICENSE.txt\", \"playwright/LICENSE.txt\"),\n]\n\ndef main():\n    records = []\n    for repo, rev, path, target in FILES:\n        url = f\"https://raw.githubusercontent.com/{repo}/{rev}/{path}\"\n        record = {\"repository\": f\"https://github.com/{repo}\", \"revision\": rev, \"upstreamPath\": path, \"url\": url, \"localSource\": f\"private/{target}\"}\n        try:\n            with urlopen(Request(url, headers={\"User-Agent\": \"IonBin-source-evaluation\"}), timeout=25) as response:\n                content = response.read(2_000_001)\n            if len(content) > 2_000_000:\n                raise ValueError(\"Source exceeds 2 MB limit\")\n            dest = ROOT / \"private\" / target\n            dest.parent.mkdir(parents=True, exist_ok=True)\n            dest.write_bytes(content)\n            record.update(bytes=len(content), sha256=hashlib.sha256(content).hexdigest(), status=\"retrieved\")\n            print(f\"Retrieved {target}: {len(content)} bytes\")\n        except Exception as exc:\n            record.update(status=\"unavailable\", error=type(exc).__name__)\n            print(f\"Unavailable {target}: {type(exc).__name__}\")\n        records.append(record)\n    (ROOT / \"source-manifest.json\").write_text(json.dumps(records, indent=2), encoding=\"utf-8\")\n\nif __name__ == \"__main__\":\n    main()\n",
      "sha256": "d6ecb127b0b6cbb09f0343d5b1d73cc6dace8b9df1b91963222fc97c0e0892f3",
      "originalExecutedSourceSha256": "d6ecb127b0b6cbb09f0343d5b1d73cc6dace8b9df1b91963222fc97c0e0892f3",
      "adaptations": [],
      "executionStatus": "Identical to original executed/authored source; not rerun during bundle preparation."
    },
    {
      "path": "viewer_probe.mjs",
      "language": "javascript",
      "content": "// Harmless browser-only marker probe. All requests are intercepted and aborted.\n// Exact upstream HTML stays in private fixtures; this harness is original IonBin code.\nimport fs from 'node:fs/promises';\nimport path from 'node:path';\nimport { fileURLToPath } from 'node:url';\nconst root = path.dirname(fileURLToPath(import.meta.url));\nconst { chromium } = await import('playwright-core');\nconst env = {};\nfor (const key of ['SystemRoot', 'WINDIR', 'COMSPEC', 'ProgramFiles', 'ProgramFiles(x86)']) if (process.env[key]) env[key] = process.env[key];\nenv.TEMP = path.join(root, 'fixtures', 'temp');\nenv.TMP = env.TEMP;\nenv.HOME = path.join(root, 'fixtures', 'home');\nenv.USERPROFILE = env.HOME;\nenv.LOCALAPPDATA = path.join(env.HOME, 'AppData', 'Local');\nenv.APPDATA = path.join(env.HOME, 'AppData', 'Roaming');\nawait fs.mkdir(env.LOCALAPPDATA, { recursive: true });\nawait fs.mkdir(env.APPDATA, { recursive: true });\nprocess.env.TEMP = env.TEMP;\nprocess.env.TMP = env.TMP;\nconst results = { mode: 'actual-generated-HTML-in-fresh-browser', reviewedOn: '2026-10-18', controls: ['Fresh browser context without accounts', 'All requests intercepted and aborted', 'Browser offline', 'Payload only sets an in-memory marker; no data access or exfiltration'], cases: [] };\nconst browser = await chromium.launch({ channel: 'chrome', headless: true, env, timeout: 15000 });\ntry {\n  results.browser = browser.version();\n  for (const name of ['benign', 'script-probe']) {\n    const context = await browser.newContext({ offline: true, serviceWorkers: 'block' });\n    const blocked = [];\n    await context.route('**/*', route => { blocked.push(new URL(route.request().url()).hostname); return route.abort(); });\n    const page = await context.newPage();\n    const errors = [];\n    page.on('pageerror', error => errors.push(error.name));\n    const html = await fs.readFile(path.join(root, 'fixtures', 'viewer', `${name}.html`), 'utf8');\n    await page.setContent(html, { waitUntil: 'load', timeout: 10000 });\n    const observed = await page.evaluate(() => ({ marker: globalThis.__IONBIN_PROBE__ === 1, outputVisible: document.body.textContent.includes('Synthetic evaluation output') }));\n    results.cases.push({ id: `viewer-browser-${name}`, skill: 'anthropic-skill-creator', mode: results.mode, expectation: 'Render synthetic evaluation data without executing it', outcome: observed.marker ? 'finding' : 'pass', observed: { ...observed, blockedRequests: blocked, browserErrors: errors }, networkRequestsAllowed: 0 });\n    await context.close();\n  }\n} finally {\n  await browser.close();\n  await fs.writeFile(path.join(root, 'browser-results.json'), JSON.stringify(results, null, 2));\n}\nconsole.log(JSON.stringify(results));\n",
      "sha256": "b3b24762bdc1a06a6fde5e55f0bf3873bbf125b6e59601efe90603cd20bcc39a",
      "originalExecutedSourceSha256": "5e2de629f25e80c60328811a0c7ebcc939d211c1bb70a7a38fdad0232807aa26",
      "adaptations": [
        "Resolve playwright-core from the local npm installation, instead of the original adjacent site dependency."
      ],
      "executionStatus": "Portable dependency lookup changed; syntax checked only, not rerun."
    },
    {
      "path": "finalize_report.py",
      "language": "python",
      "content": "\"\"\"Build sanitized public evidence summaries from local controlled probe results.\"\"\"\nimport hashlib\nimport json\nfrom pathlib import Path\n\nROOT = Path(__file__).resolve().parent\ndata = json.loads((ROOT / 'results.json').read_text(encoding='utf-8'))\nbrowser = json.loads((ROOT / 'browser-results.json').read_text(encoding='utf-8'))\ncases = [case for suite in data['suites'] for case in suite.get('cases', [])] + browser['cases']\nfor case in cases:\n    case['skillId'] = case.pop('skill', case.get('skillId'))\n    case['passed'] = case['outcome'] == 'pass'\n    case['failed'] = case['outcome'] == 'finding'\n\ndata['environment'].update(node='24.18.0', chrome=browser['browser'], poppler='26.07.0', gitBash='5.3.15(1)-release (x86_64-pc-cygwin)')\nrevision = '34040c9c568585f6929bedeaad110ad08f079624'\nviewer_source = f'https://github.com/anthropics/skills/blob/{revision}/skills/skill-creator/eval-viewer/generate_review.py'\ndefinitions = {\n    'anthropic-pdf': {\n        'mode': 'actual-helper-probes',\n        'findings': [\n            {'severity':'attention','kind':'observed-data-handling','title':'Submitted values appear in diagnostics','detail':'The exact helper echoed SYNTHETIC_VALUE_MARKER for an invalid checkbox value. This confirms diagnostic behavior, not a remotely exploited vulnerability. Treat logs with the document data\\'s confidentiality.','testIds':['pdf-diagnostic-value']},\n            {'severity':'attention','kind':'robustness-limit','title':'Non-form PDF raises AttributeError','detail':'Calling get_field_info on a valid synthetic PDF without AcroForm fields raised AttributeError instead of returning an empty list. This failed an added robustness expectation; it does not imply form-filling failure on supported inputs.','testIds':['pdf-no-form']},\n        ],\n        'limitations':['No end-to-end agent task or skill-vs-baseline effectiveness comparison','No OCR, redaction, encryption, hostile PDF parser exploit or large-resource tests','No rendered visual-quality review; raster output was checked for page count and dimensions','Only the three named helpers and these synthetic fixtures were executed','Proprietary/source-available helper code is retained privately and excluded from public artifacts'],\n    },\n    'anthropic-docx': {\n        'mode':'actual-helper-probes', 'findings':[],\n        'limitations':['Only merge_runs and its safe_extract helper were exercised','No full DOCX rendering, Word/LibreOffice interoperability, tracked-change acceptance or generated-document quality tests','Nominal merge probes used unpacked synthetic document XML; traversal used a synthetic ZIP','No ZIP bombs, arbitrary hostile native code or large-resource tests','Proprietary/source-available helper code is retained privately and excluded from public artifacts'],\n    },\n    'anthropic-skill-creator': {\n        'mode':'actual-viewer-helper-plus-browser-marker-and-mocked-effects',\n        'findings':[{'severity':'high','kind':'confirmed-in-pinned-revision','title':'Evaluation text can escape the viewer script context','detail':'At revision '+revision+', generate_html embeds json.dumps output directly in an inline script. Synthetic prompt text with a closing script tag executed a harmless in-memory marker in fresh offline Chrome. The benign control did not execute the marker. This is confirmed only for the pinned files and fixture; current upstream HEAD was not tested. Two failed expectations describe this one issue.','sourceUrl':viewer_source+'#L279','testIds':['viewer-script-boundary','viewer-browser-script-probe'],'mitigation':'Treat evaluation text as untrusted; use a script-safe JSON serialization or non-executable data container with correct escaping before viewing untrusted outputs.'}],\n        'limitations':['No Claude subprocess, live model, API credential, optimization loop or generated-skill effectiveness evaluation','Port-cleanup lsof/os.kill effects were mocked; no real process was terminated','The browser payload only set a marker; no sensitive data or exfiltration was involved','All browser requests were blocked, including attempted Google Fonts and SheetJS requests','The viewer may need those external resources for fonts or XLSX rendering; their downloaded code was not evaluated'],\n    },\n    'openai-playwright': {\n        'mode':'mocked-wrapper-dispatch-only', 'findings':[],\n        'limitations':['Five wrapper dispatch checks used a fake npx, plus one Git Bash environment check','No real npx, npm package, Playwright CLI or target browser workflow was executed','No install/compatibility, live-site, account, session-security or artifact-confidentiality claim follows','The wrapper still delegates to an unpinned npm package; these checks do not evaluate that package','This is the historical catalog revision, not the current successor plugin'],\n    },\n}\nreports=[]\nfor skill_id, definition in definitions.items():\n    selected=[case for case in cases if case['skillId']==skill_id]\n    upstream_prefix={'anthropic-pdf':'skills/pdf/','anthropic-docx':'skills/docx/','anthropic-skill-creator':'skills/skill-creator/','openai-playwright':'skills/.curated/playwright/'}[skill_id]\n    sources=[item for item in data['upstreamSources'] if item['upstreamPath'].startswith(upstream_prefix)]\n    report={'schemaVersion':1,'skillId':skill_id,'reviewedOn':'2026-10-18','status':'bounded-helper-evaluation','mode':definition['mode'],\n            'revision':sources[0]['revision'],'total':len(selected),'passed':sum(case['passed'] for case in selected),'failed':sum(case['failed'] for case in selected),\n            'countMeaning':'Counts are individual expectations, not vulnerability counts or skill-effectiveness scores. Mocked effects are explicitly labeled.',\n            'environment':data['environment'],'commands':[data['command']]+(['node viewer_probe.mjs'] if skill_id=='anthropic-skill-creator' else []),\n            'controls':data['controls'],'isolationLimits':data['isolationLimits'],'findings':definition['findings'],'limitations':definition['limitations'],\n            'cases':selected,'sources':sources,'repetitions':1}\n    reports.append(report)\n    (ROOT/f'{skill_id}.json').write_text(json.dumps(report,indent=2),encoding='utf-8')\ndata['browserProbe']=browser\ndata['reports']=reports\ndata['summary']={'total':len(cases),'passed':sum(case['passed'] for case in cases),'failed':sum(case['failed'] for case in cases),'distinctAdverseObservations':3,'countMeaning':'Four failed expectations cover three distinct observations; two failures confirm the same viewer issue.'}\ndata['artifactHashes']=[]\nfor relative in ['pdf/form.pdf','pdf/nominal.pdf','pdf/images/page_1.png','docx/nominal/word/document.xml','viewer/benign.html','viewer/script-probe.html']:\n    source=ROOT/'fixtures'/relative\n    data['artifactHashes'].append({'syntheticArtifact':relative,'bytes':source.stat().st_size,'sha256':hashlib.sha256(source.read_bytes()).hexdigest(),'publication':'Private fixture; reproducible from the harness. Upstream-derived HTML is not included in public reports.'})\ndata['priorAttempts']=[\n    {'step':'source acquisition','result':'Default sandbox network unavailable; scoped public-source download succeeded.'},\n    {'step':'dependency acquisition','result':'Combined automatic approval review timed out; one narrow retry succeeded.'},\n    {'step':'DOCX dependency path','result':'helpers.py was unavailable; verified package entrypoint helpers/__init__.py retrieved at the same commit.'},\n    {'step':'Git Bash and Chrome startup','result':'Default sandbox blocked runtime startup; reviewed, scoped supervised execution succeeded. No OS-isolation claim made.'},\n]\n(ROOT/'results.json').write_text(json.dumps(data,indent=2),encoding='utf-8')\n\nlines=['# Controlled helper evaluations','', 'Review date: 2026-10-18. These are bounded code/helper probes, not a full evaluation of agent behavior or a safety certification.','',\n       '| Skill | Expectations | Passed | Failed | Execution scope |','|---|---:|---:|---:|---|']\nfor report in reports:\n    lines.append(f\"| {report['skillId']} | {report['total']} | {report['passed']} | {report['failed']} | {report['mode']} |\")\nlines+=['','**29 expectations: 25 passed, 4 failed.** The failures represent three distinct observations; two checks confirm the same viewer issue. Five Playwright checks used fake executable dispatch, plus one Bash environment check. They do not establish CLI or browser compatibility.','',\n        '## Observed findings','',\n        f'**High \u2014 viewer script-context injection, pinned revision only.** The skill-creator generator inserts raw JSON into an inline script. A synthetic prompt containing a closing script tag executed a harmless in-memory marker in fresh offline Chrome; the benign control did not. No live endpoint, sensitive content or exfiltration was used. Current upstream HEAD was not tested. [Affected source]({viewer_source}#L279).','',\n        '**PDF diagnostic disclosure.** Invalid-checkbox output echoed the synthetic submitted value. This confirms a log-handling concern; it is not evidence of a remote exploit.','',\n        '**PDF robustness limit.** The form-field extractor raised AttributeError for a valid PDF without form fields. This was an added boundary expectation, not a claim that supported form inputs fail.','',\n        '## Actual operations and controls','',\n        'PDF probes discovered two form fields, filled text and checkbox values, preserved the source, rejected invalid field/page data and malformed PDF, and rasterized one page to 400 \u00d7 266 pixels. DOCX probes checked eligible merges, whitespace, formatting/revision boundaries, malformed/entity XML and parent-path ZIP rejection. The viewer generated synthetic reports; cleanup process dispatch was mocked.','',\n        'Python 3.12.10; Windows 11 build 26200; pypdf 6.19.0; ReportLab 4.4.9; Pillow 12.3.0; pdf2image 1.17.0; defusedxml 0.7.1; Poppler 26.07.0; Git Bash 5.3.15; Chrome '+browser['browser']+'; Node 24.18.0.','',\n        'Fresh child environments excluded account/credential variables. Inputs were synthetic, Python writes were guarded to fixture directories, Python sockets were disabled, and child runs had timeouts. Browser requests were intercepted and blocked; Google Fonts and SheetJS attempted loads were observed, with zero requests allowed. No real process termination, npm CLI execution or authenticated account action occurred.','',\n        '**This was supervised local execution, not an OS sandbox.** Docker was unavailable and WSL was not installed. Application guards and known fixtures do not establish containment of arbitrary hostile code. No memory quotas, ZIP bombs, large-resource attacks or hostile native-code tests were attempted.','',\n        '## Reproduction and evidence','',\n        'Acquire only the pinned source list with `python fetch_sources.py`. Keep `private/` and `fixtures/` excluded from publication. Install defusedxml 0.7.1 into `private/deps` if unavailable. Provide the existing runtime library and Poppler locations to `python run_evals.py --runtime-libs <site-packages> --poppler-bin <poppler-binaries>`. Run `node viewer_probe.mjs`, then `python finalize_report.py`. The browser harness expects the already-installed adjacent site Playwright dependency.','',\n        'Each per-skill JSON includes exact source URLs and SHA-256 hashes, commands, observations, failed expectations, environment, limitations and license-file references. `results.json` also records synthetic artifact hashes. Source/helper files and generated upstream-derived HTML remain private. Proprietary PDF/DOCX source is not part of public reports.','',\n        'A single deterministic pass per case was recorded. No effectiveness improvement, broad reliability, current-version safety or full-skill runtime certification is claimed.']\n(ROOT/'report.md').write_text('\\n'.join(lines)+'\\n',encoding='utf-8')\nprint(json.dumps(data['summary']))\n",
      "sha256": "d4e5e482fdabca25459e7569431e5a412a0e731178bc1f20ecfde6879eab6231",
      "originalExecutedSourceSha256": "d4e5e482fdabca25459e7569431e5a412a0e731178bc1f20ecfde6879eab6231",
      "adaptations": [],
      "executionStatus": "Identical to original executed/authored source; not rerun during bundle preparation."
    }
  ]
}