Operator brief · 170

Reproducibility is what turns a run into evidence.

The key idea

The problem with comparing runs

Two simulations of the same system disagree by design.

Monte Carlo output is stochastic, which means two runs of an identical configuration will produce different numbers. Usually the difference is small; occasionally it is large enough to look like a finding. This is the trap that catches careful people: change one exit parameter, re-run, observe an improvement, and conclude the parameter helped. The improvement may be entirely sampling noise, and nothing about the output announces which it was. Without a control for randomness, every research comparison in the workbook carries an unmeasured error term, and the operator's judgement about what counts as a meaningful difference is doing work it is not equipped to do.

The control

Fix the seed, and the only remaining difference is the one you made.

A fixed random seed makes a run exactly reproducible — the same seed with the same configuration yields the same paths, every time. The useful consequence is the one the manual states directly: the same seed with different structures gives a like-for-like comparison. The two runs face identical sequences of luck, so any divergence in outcome is attributable to the structural change rather than to the draw. This is the difference between an experiment and an observation. It costs one setting, it is available on every run, and skipping it converts the most common research activity in the Lab — testing whether a change is an improvement — into a coin flip that reports to three decimal places.

FigureWhat each seed-and-structure pairing can honestly answer

Structure	Same seed	Different seed
Unchanged	Reproduction check — output must match exactly	Stability check — how much is noise alone?
Changed	The experiment — difference is attributable	Uninterpretable — two causes, one number

Two dials, four combinations, and only two of them answer a question worth asking. The diagonal cases are where most informal research accidentally lives.

The other diagonal

Deliberately varying the seed is a test in its own right.

Holding the seed constant answers whether a change did something. Varying it, with the structure held still, answers something equally important: how much the result moves for no reason at all. That figure is the noise floor, and it is the only honest yardstick for deciding whether an observed improvement is worth anything. A change that shifts completion probability by less than the spread between identical configurations under different seeds has not been shown to do anything. Running both diagonals — reproduce, then perturb — costs two extra executions and replaces the entire category of research findings that evaporate on contact with a second look.

The registry

One row per execution, appended, never overwritten.

The run log records each execution with its identity, seed, scenario, and headline results. It is an append-only history rather than a status display, and the distinction is the whole point: it preserves the runs that were disappointing alongside the ones that were quoted. Research memory is otherwise ruthlessly selective — the encouraging run gets screenshotted into a review, the twelve that preceded it are simply not mentioned, and nobody involved is being dishonest. A registry that accumulates every execution makes the denominator visible. If a finding required nine attempts to appear, the log says so, and that fact is usually more informative than the finding.

What graduation requires

Repeat across seeds, survive adjacent values, hold in recent evidence.

The page above states the criterion for leaving the sandbox, and seed discipline is what makes it checkable rather than rhetorical. A candidate finding must repeat when the draw changes, which the registry can demonstrate. It must survive adjacent parameter values, which means the neighbouring configurations were also run and also logged — a result that holds at one setting and collapses at the next one over is a cliff edge, not a structure. And it must hold in the most recent evidence window, since a finding that only appears against older calibration is a statement about a period rather than about the system. Three conditions, all of them evidenced by runs that exist in the log.

The key idea

A sandbox without reproducibility is not research — it is browsing.

The freedom to test anything is only valuable if the tests produce claims that survive scrutiny, and survival requires that a second person, or the same person in six months, can regenerate the result and get the same answer. Seed control supplies that. The registry supplies the history that stops selective memory from doing the summarising. Neither is glamorous, and together they are what separates a research layer that genuinely feeds the promotion workflow from one that generates confident opinions at a rate nobody can audit.

Connected inside MARS

R&D Boundary →

Scenario Manager →

Quality Assurance →

Every brief documents the same shipped system.

The complete MARS package — eleven workbooks, three TradingView indicators, the full manual library — $497.

Get MARS Pro — $497 Explore the Platform