How to Fingerprint AI Models When Prompts Lie

When evaluating third-party API gateways, proxy routers, or anonymous arena models, prompt-based identification is essentially useless. A basic system prompt or lightweight fine-tune can make Claude claim it is GPT-4, make Llama sound like Mistral, or alter refusal behaviors entirely.

Persona and tone are malleable. But the underlying serving stack (tokenizers, merge tables, serving harnesses, and error validation boundaries) is rigid. To fake those, a proxy provider would need to intercept streams, translate tokens bidirectionally in real time, and rewrite metadata on the fly, which introduces severe latency and stream desyncs.

Here is how our client-side fingerprinting engine identifies models deterministically using infrastructure artifacts instead of conversational cues.

1. Tokenizer Vocabularies and Byte Fallbacks

Every lab trains a distinct tokenizer with custom vocabulary sizes, merge tables, and fallback rules. Because the tokenizer determines the exact input matrix shape of the model weights, a model cannot alter how it segments text without a full retrain.

We send small, fixed text inputs to the endpoint and inspect the returned usage.prompt_tokens:

2. Template Offsets and Hidden Harnesses

Inference engines (vLLM, SGLang, TGI, TensorRT-LLM) wrap user prompts in chat templates (like ChatML or Jinja templates) and often inject default system prompts.

By sending an empty prompt or a single 1-token payload and comparing the returned usage.prompt_tokens against the raw payload token count, we calculate the static overhead:

This delta exposes hidden guardrail templates, platform wrappers, and chat formatting artifacts applied upstream before the prompt hits the model.

3. Error Taxonomy and Validation Ceilings

Validating normal responses only tells part of the story. Feeding intentionally invalid parameters triggers backend validation written by specific engineering teams:

• Temperature ceilings: Sending temperature: 2.5 causes some runtimes to fail at >1.0, others at >2.0, with distinct error strings.
• Context refusals: Requesting max_tokens: 1000000000 forces the backend to reject the request and echo its true physical generation limit (e.g. "max_tokens must be ≤ 131072").
• Vendor error codes: Safety filter triggers often return proprietary numeric codes (e.g. Zhipu AI internal safety code 1301).

4. Response Serialization Dialects

Minor implementation details in how the server formats JSON reveal the runtime:

• The exact choices[0].finish_reason string (e.g. stop vs end_turn vs content_filter).
• How null or optional fields are serialized in the response body.
• Server-Sent Events (SSE) framing quirks during streaming.

Summary of Probe Vectors

Client-Side Execution

Because probes are purely deterministic, we run them directly in the browser. Requests are dispatched straight from the client tab to the target endpoint without passing through any intermediate proxy or logging server.

Each probe exports a simple runner contract that returns a deterministic value:

export default {
  name: "tokenizer/cjk-density",
  description: "Measures CJK token segmentation count",

  async run(ctx) {
    const res = await ctx.chat({
      messages: [{ role: "user", content: "人工智能模型基准评测体系" }],
      max_tokens: 1,
      temperature: 0,
    });

    return {
      value: res?.usage?.prompt_tokens ?? "ERR_NO_USAGE",
    };
  },
};

Why This Matters

As API wrappers and proxy routers become more prevalent, verification cannot rely on trust or conversational vibes. By testing deterministic infrastructure artifacts like tokenizers, template deltas, and boundary conditions, we can accurately verify what model is actually serving your traffic.