Objective conditioning: five rounds, five forms, no effect

The claim and its five forms

In short: Five ways of letting the objective shape the index, five negative results.

Round 1: the thesis fails

In short: Corpus conditioning is worth a great deal; adding the objective subtracts from it.

The conditioning ladder varies only what the designer is shown. Every rung shares an encoder, a builder, and an index.

The cross-over test asks whether a taxonomy designed for objective A serves A's workload better than one designed for B, and vice versa. It found nothing: +0.020 (p=0.35) in one direction and −0.002 (p=0.92) in the other. Taxonomies designed under different objectives served each other's workloads indistinguishably. A sampler test in the same round, which stratified the designer's sample across discovered topics so that minority material could not vanish, did what it was designed to do (minority-topic coverage of the sample rose from 60 to 100 percent) and bought nothing in retrieval (−0.015, p=0.70); worse, it produced a significant gain on the balanced control corpus where the preregistration said the gap should vanish. An effect that appears where it should not and is absent where it should be is not the claimed mechanism.

An instruction-conditioned baseline in this round was unsound and is recorded as such: the encoder used fixed prefixes rather than following instructions, so prepending the objective to each query merely diluted it. That comparison measured query dilution, not instruction conditioning, and its nominal result is not used anywhere.

Rounds 2 and 3: the pipeline as specified, and a fragile cross-over

In short: With the gate, guided topics, and noise cleaning restored, the cross-over appeared at two of three operating points on one machine, and at none of three on a second.

Round 1 had tested a stripped-down reconstruction: no relevance gate, no model guidance of the topic model, no model cleaning of clustering noise. Each of those plausibly explained the failure, since under partition-mode carding every span must land in some view, and a taxonomy aimed at three departments had to absorb the seven it was not aimed at. Round 2 restored all three, applied the gate identically to every arm, and scored both taxonomies over a common per-workload pool (scoring each over its own gated pool had let the gate carry the comparison, halving the effect when corrected). Round 3 repaired a degenerate topic model (3 topics against 12) that had been diagnosed in round 2's own metrics before any attempt to fix it.

The A direction is robust everywhere and grows as the gate loosens. The B direction is fragile. Replicated on a second machine at the same commit, with the encoder on the CPU and a fixed hash seed, the B direction read −0.018 (p=0.32), +0.007 (p=0.65) and −0.040 (p=0.034, the wrong direction): zero of three. The mechanism is understood: dimensionality reduction and density clustering produce different topic models on a different processor architecture, the designer sees different samples and writes different view sets, and the B-direction effect depends on which view set it happens to write. The ladder over the same sweep read +0.019, −0.061 (p=0.0018) and −0.012: it changed sign across operating points, so the single-workload comparison cannot be quoted in either direction.

What rounds 2 and 3 established is narrower than the claim: conditioning produces genuinely different taxonomies, and in one direction each serves its own workload better. It did not establish that an objective-designed taxonomy beats a well-formed corpus-derived one on any workload.

Round 4: more context, worse retrieval, and the stopping rule

In short: Business context and schema metadata, the conditioning signals an enterprise actually has, made retrieval worse in proportion to how much was supplied.

The maintainer's argument for this round was that enterprises do not organise data for one question; they organise a line of business's data so that several agents can ask many questions, and the conditioning signal available in practice is standing business context plus schema metadata rather than a narrow objective. The workload was built accordingly: 200 queries across six departments. The round was preregistered before implementation with a stopping rule: if context plus metadata did not beat the corpus sample alone, the programme would stop pursuing the conditioning claim.

The taxonomies show why. From the corpus alone the designer derived six views that followed the data (fault logs, system maintenance, financial risk, legal compliance, logistics and supply, IT incidents). Given the business narrative it produced four elegant views organised around the business (commercial agreements, operations and logistics, finance reporting, risk and compliance). The business framing covers the corpus less well, and on a workload spanning six departments coverage is what matters. The taxonomies were genuinely different (mean pairwise similarity 0.833, so the convergence condition did not fire) and genuinely worse. The stopping rule fired and the programme ceased pursuing the conditioning claim in that form.

Round 5: the objective enters late, on real data

In short: Embedding once and selecting views per objective from a shared, unseeded topic substrate is the practical version of the idea. It does not carry the effect, and on human-judged data it is the worst arm.

The maintainer's final architecture separated two things that rounds 2 to 4 had conflated: conditioning the topic model on the objective (seed topics written from it) and conditioning the view design on it. In round 5 the corpus is embedded once and its topics discovered once with no seeding, a shared substrate reflecting the corpus alone; the objective enters only later, when a model selects and composes a view set from that substrate, and at the per-objective gate. The economic appeal is real: a new objective then costs a few model calls, not a new embedding pass. Preregistration was waived for this round at the maintainer's direction; rigour was held by fixing the decision rules in code before any number was read, requiring every verdict to hold across several topic-model seeds, and reporting every arm including the ones that beat the method.

On the synthetic corpus, with the validated card builder and five topic-model seeds, the two-sided cross-over held at one seed of five. The A direction was robustly positive (+0.034 to +0.099, significant in four of five); the B direction ranged from −0.038 to +0.025. The cross-over of round 3 had lived in the objective seeds, not in the idea; the practical embed-once version does not carry it.

FollowIR supplies the real test: news collections where each query comes with an instruction and two sets of human judgements, before and after the instruction changes, so an index that follows the objective can be told apart from one that ignores it. Five arms were scored: one pooled vector per document; the same encoder with the instruction prepended to the query (a weak instruction-conditioned baselineInstruction-following retriever. An embedding model trained so that a stated instruction (for example, only reports from outside the UK) changes what it ranks as relevant, instead of only the query words., since the encoder was not trained to follow instructions); matched blind chunks; a single objective-blind view set derived from the corpus; and the view set a model selected per instruction from the shared substrate.

Two caveats were recorded with the result, and both point the same way. The encoder is a weak instruction baseline; a retriever trained to follow instructions would widen the gap against the method, not narrow it. And the candidate pool depth and document length were capped for speed, identically across every arm. A third collection (Robust04) was not run once the direction was consistent across the synthetic test and two real collections; it would have added corroboration, not information.

What this establishes

Independently measured Synthetic results across seeds and machines; FollowIR with human judgements.