{
  "measured_by": "GetIntel",
  "measured_on": "2026-08-11",
  "question": "How many reruns does an AI visibility number need before it means anything?",
  "method": "Every prompt-and-engine series with at least 10 runs was split into two groups: series where our brand's cited/not-cited state never changed, and series where it did. For the varying series we took the running estimate after k runs and measured its absolute error against that series' final rate over all its runs.",
  "scope": {
    "series_with_10plus_runs": 320,
    "series_that_never_vary": 275,
    "series_that_vary": 45,
    "pct_that_never_vary": 85.9,
    "runs_per_series_range": "11 to 12"
  },
  "on_varying_series": {
    "note": "Error is in percentage points of the cited rate. A single run can only ever return 0% or 100%, which is why its error is large.",
    "by_number_of_runs": [
      {
        "runs": 1,
        "mean_absolute_error": 0.327,
        "pct_within_10_points": 31.1,
        "pct_within_20_points": 44.4
      },
      {
        "runs": 2,
        "mean_absolute_error": 0.255,
        "pct_within_10_points": 31.1,
        "pct_within_20_points": 48.9
      },
      {
        "runs": 3,
        "mean_absolute_error": 0.24,
        "pct_within_10_points": 31.1,
        "pct_within_20_points": 48.9
      },
      {
        "runs": 4,
        "mean_absolute_error": 0.208,
        "pct_within_10_points": 35.6,
        "pct_within_20_points": 57.8
      },
      {
        "runs": 5,
        "mean_absolute_error": 0.185,
        "pct_within_10_points": 40.0,
        "pct_within_20_points": 62.2
      },
      {
        "runs": 6,
        "mean_absolute_error": 0.163,
        "pct_within_10_points": 48.9,
        "pct_within_20_points": 68.9
      },
      {
        "runs": 7,
        "mean_absolute_error": 0.123,
        "pct_within_10_points": 55.6,
        "pct_within_20_points": 86.7
      },
      {
        "runs": 8,
        "mean_absolute_error": 0.085,
        "pct_within_10_points": 68.9,
        "pct_within_20_points": 88.9
      },
      {
        "runs": 9,
        "mean_absolute_error": 0.06,
        "pct_within_10_points": 82.2,
        "pct_within_20_points": 100.0
      },
      {
        "runs": 10,
        "mean_absolute_error": 0.036,
        "pct_within_10_points": 100.0,
        "pct_within_20_points": 100.0
      }
    ]
  },
  "headline": "On a prompt whose answer actually varies, one check is off by 32.7 percentage points on average and lands within 10 points only 31.1% of the time. Seven runs cut the error to 12.3 points.",
  "the_catch": "275 of 320 series, 85.9%, never varied at all. For those, one run is the whole truth and nine more add nothing. You cannot tell which group a prompt belongs to without having already run it repeatedly, which is the actual argument for a schedule.",
  "important_limit": "Our series run 11 to 12 deep, so the 'final rate' each estimate is measured against is itself built from 11 or 12 runs. At k=9 and k=10 the estimate shares most of its data with the target, so the apparent convergence there is partly definitional rather than evidence that 10 runs is sufficient. Read the k=1 to k=7 trend, which is not contaminated that way; treat the k=9 and k=10 rows as a ceiling artefact.",
  "limits": "One brand, one category, four engines, 11-12 runs per series over 10 July to 10 August 2026. 45 varying series is a small group.",
  "false_movement_test": {
    "method": "Each series with 10+ runs was split into its first and second half and the cited rate compared. Nothing happened between the halves: no publishing, no campaign, no change of any kind. Any difference is pure noise, so this measures how often a before-and-after comparison invents an improvement that is not there.",
    "series": 320,
    "by_threshold": [
      {
        "points": 5,
        "moved_up": 37,
        "moved_down": 6,
        "pct_either": 13.4
      },
      {
        "points": 10,
        "moved_up": 36,
        "moved_down": 6,
        "pct_either": 13.1
      },
      {
        "points": 20,
        "moved_up": 21,
        "moved_down": 5,
        "pct_either": 8.1
      },
      {
        "points": 30,
        "moved_up": 20,
        "moved_down": 3,
        "pct_either": 7.2
      }
    ],
    "on_the_45_varying_series": [
      {
        "points": 10,
        "pct_either": 93.3
      },
      {
        "points": 20,
        "pct_either": 57.8
      },
      {
        "points": 30,
        "pct_either": 51.1
      }
    ],
    "finding": "Across all 320 series, 13.1% showed a swing of 10 points or more between halves with nothing having changed. On the 45 series that vary at all, 93.3% did. So on a prompt known to be unstable, a 10-point before-and-after difference is almost meaningless on its own.",
    "asymmetry_caveat": "Moves were not symmetric: at the 10-point threshold 36 series drifted up and 6 drifted down. Part of that is real - our citation count genuinely grew over the window as pages were published - so this test overstates pure noise in the upward direction and the downward figures are the cleaner read of it."
  }
}