Hmm… this already seems to have been built quite carefully, so I am not sure there is much more I can add at this point. Still, I tried a few bounded experiments in Colab:
I agree with the point above that validation/rollback and the fresh-holdout design seem like the highest-leverage places to focus.
My overall impression is not that the RFC needs a broader safety claim. The current Draft 7B RFC, Campaign governance document, and retained 7B showcase already seem unusually careful about distinguishing feasibility evidence from stronger claims.
The most useful addition I can see would be to make a few state transitions more explicit:
- a quorum aggregate is a candidate, not automatically the next canonical checkpoint;
- artifact reconstruction is separate from task-quality evaluation;
- ordinary restart recovery is separate from a trust-changing corrective rollback;
- development evidence is separate from terminal confirmation;
- named roles are separate from presently accepted human capacity.
A possible default path would be:
accepted deltas
→ immutable aggregate candidate
→ structural and inventory checks
→ capture a deterministic in-process candidate reference
→ reconstruct the native PEFT artifact in a fresh process
→ check round-trip parity
→ check Adapter-enabled vs Adapter-disabled behavior
→ atomically promote Adapter + matching outer state
→ run task evaluation only on the promoted canonical checkpoint
The two behavioral checks answer different questions:
- Round-trip parity: did the fresh process reconstruct the candidate that the aggregation process intended?
- Activation contrast: is the reconstructed Adapter actually active and producing the expected bounded effect relative to Adapter-disabled execution?
Enabled-versus-disabled alone would not establish round-trip parity: an Adapter could be active while still being reconstructed under a numerically different runtime lane.
The failure path could remain conservative:
structural mismatch,
round-trip parity failure,
activation failure,
or unresolved result
→ quarantine the candidate
→ leave the canonical pointer unchanged
→ do not issue descendant work from it
unless a finite provisional depth was declared in advance
For a trust-changing rollback, I would separate the operation from ordinary process recovery:
pause
→ select a named trusted ancestor
→ restore Adapter + matching outer state
→ create a new canonical-transition identity
→ fence work issued from an abandoned lineage
→ preserve descendants as superseded history
→ reconcile state and resume
A monotonically increasing Campaign generation/epoch seems like one straightforward implementation, although an equivalent fencing identity could work too.
Bounded Colab observations
I ran four small checks. None of them is a general safety or compatibility proof, but together they helped narrow the transition rules above.
| Probe |
Bounded observation |
What it does not show |
| Direct 7B NF4 micro-work-unit |
A direct Qwen2.5-7B NF4 micro-work-unit completed on a Colab T4 at roughly 9.5 GiB peak reserved device memory. |
It does not establish the ordinary Volunteer path, independent-host E2E, or a sustainable Campaign cadence. |
| PEFT round-trip |
In one prepared runtime, all 392 Adapter tensors and a fixed behavioral fixture matched after native PEFT save/reload. With a different base-preparation lane, the tensors still matched but behavior changed. |
It does not establish universal PEFT portability or poisoning resistance. |
| Recovery/rollback rehearsal |
A saved v0 result was rejected after the live branch advanced to v1. Restoring the v0 ancestor backup preserved its active lease, and the same result was then accepted under the restored v0 state. |
It is not a vulnerability or a backup/restore defect. It distinguishes ordinary recovery from corrective rollback. |
| Loader/dtype rehearsal |
The same native Adapter and NF4 base produced different enabled logits when the loading policy materialized Adapter tensors in FP32 rather than FP16. Adapter-disabled logits remained identical. |
It does not show an inherent PEFT-native versus Transformers-native incompatibility. |
The last two suggest that a promotion record may need to capture, or directly verify, more than a weight-file hash:
- exact base revision;
- quantization and base-preparation lane;
- loader/injection policy;
- actual loaded Adapter dtype;
- resolved target inventory;
- fixed-fixture parity and activation results;
- matching outer-state identity.
This is consistent with PEFT’s own API distinction around autocast_adapter_dtype: successful key loading alone does not fully determine the numerical execution lane.
A compact binding model
I would not necessarily put everything into one giant manifest. Separate artifacts seem fine as long as their references reconcile unambiguously.
| Object |
Minimum binding that seems useful |
| Work unit / lease |
Campaign or amendment identity, parent canonical checkpoint, lineage generation, lease generation |
| Aggregate candidate |
Parent checkpoint, accepted work/result set, Adapter/config, matching outer state |
| Promotion record |
Candidate identity, runtime contract, structural result, parity result, activation result |
| Evaluation |
Exact promoted checkpoint, evaluation-contract identity, and evidence role |
| Critical-role acceptance |
Campaign/amendment identity, role, stable public handle, responsibility horizon |
| Ledger event |
Transition identity plus before/after state references |
Evaluation and the 256 → 1,024 decision
The retained showcase already used a useful discipline: a 128-item development set identified an over-strong attempt even though validation loss improved, and a disjoint confirmatory set was preregistered for the final attempt.
For the proposed Campaign, I would extend that distinction over time:
D: development/extension evidence used at 128 and 256;C: a separate sealed confirmation set used once for the actual terminal release or quality-claim candidate.
Then the extension decision could use three independent gates:
S = systems integrity and rollback readiness
Q = quality and harm evidence on D
H = human and resource capacity
All three should pass; one should not compensate numerically for another.
Q result
├─ Green
│ └─ extension is possible only if S and H also pass
├─ Red
│ └─ stop
└─ Gray
└─ pause; default to no extension
└─ optionally run one preregistered bounded tie-breaker,
if it is fresh and capacity allows it
This would preserve the RFC’s existing statement that a step target is not permission to continue through a failed quality or safety gate.
The human gate matters because promotion, quarantine, evaluation, incident review, and rollback all consume scarce reviewer capacity. If that capacity is unknown, I would treat it as unavailable rather than silently assuming it exists.
A compact amendment could therefore look something like this:
| RFC area |
Small possible addition |
| Delta validation |
Candidate state, fresh-process parity/activation gate, atomic promotion or quarantine |
| Rollback |
Distinguish restart recovery from corrective rollback; fence abandoned-lineage work |
| Evaluation |
Explicit D/C roles and evaluator identity |
| 128/256 rule |
128 as a kill-switch; 256 as independent S/Q/H decision |
| Governance |
Campaign-bound role acceptance, backup/handoff, capacity-based pause |
| GSM8K |
Keep the bounded primary objective; add external or perturbational audits only if capacity remains |
Probe setup, exact observations, and interpretation boundaries
1. Direct 7B NF4 micro-work-unit
The direct NF4 check established that a tiny Qwen2.5-7B LoRA work unit can complete on a Colab T4 within the observed memory envelope.
The useful conclusion is narrow:
A 16 GB-class T4 lane is not automatically impossible for a bounded direct micro-work-unit.
It does not establish:
- the ordinary Volunteer Campaign integration;
- two independent physical hosts;
- invite/HTTPS/submission/recovery/cleanup E2E;
- an appropriate minimum work-unit size;
- a sustainable promotion-gate cadence;
- acceptable contributor download, runtime, or energy cost.
The observed gate time also should not be converted directly into a Campaign policy. Promotion cadence depends on aggregate frequency, warm/cold cache behavior, candidate queues, rollback exposure, and human review capacity.
2. Native PEFT round-trip
The round-trip check used the retained 392-tensor standard PEFT Adapter shape.
Within one prepared runtime:
- tensor inventory matched;
- tensor values matched;
- the fixed behavioral fixture matched.
When the same tensor artifact was loaded under a different base-preparation lane:
- tensor identity still matched;
- behavior differed.
That suggests the executable artifact contract is closer to:
Adapter payload and config
+ exact base revision
+ base preparation and quantization
+ tokenizer/template where relevant
+ loader/injection policy
+ actual loaded dtype and resolved targets
+ bounded behavioral evidence
rather than the weight file alone.
This remains a bounded compatibility observation, not a universal rule that every package in the environment must be permanently pinned. A practical split might be:
- retain a broad environment inventory for diagnosis;
- treat only demonstrated execution-critical fields as identity-critical;
- verify the rest through the fresh-process behavioral gate.
3. Corrective rollback fencing rehearsal
This probe used CrowdTensor main at commit ed76a2bece39f6cdf6c9e862f5f96173500b8f1d.
A tiny synthetic Campaign was created at Adapter v0, with one active lease preserved in a backup.
The sequence was:
create v0 lease
→ save ancestor backup
→ complete the round and advance the live state to v1
→ submit the saved v0 result to the live v1 state
→ observe volunteer_stale_adapter_version_rejected
→ restore the v0 ancestor backup
→ recover with active lease preserved
→ submit the same saved v0 result
→ observe acceptance under restored v0 state
This matches the current code’s two ordinary behaviors:
- older Adapter versions are rejected on the advanced live branch;
- restart recovery preserves valid active leases.
Relevant implementation points are visible in the current Coordinator:
Those are coherent restart-recovery semantics. The additional design question is what happens when the Operator is not merely restarting the same trusted lineage, but abandoning a previously canonical branch.
A generation/epoch decision tree could be:
Where was the problem found?
├─ Before candidate promotion
│ └─ quarantine the candidate;
│ no canonical rollback occurred
└─ After canonical promotion
└─ did descendant or in-flight work bind to that state?
├─ yes or unknown
│ └─ increment lineage generation
│ and reject old-generation work
└─ no
└─ an audited pointer correction may be enough,
although always incrementing generation is simpler
The rollback unit should include the matching training state, not just Adapter weights:
Adapter
+ adapter configuration/runtime identity
+ outer velocity or optimizer state
+ outer step
+ trusted parent checkpoint
The current aggregation path already writes Adapter, config, and outer velocity together before advancing adapter_version, outer_step, and the current pointers, so this seems compatible with the existing design rather than a separate architecture: current aggregation transition.
Historical descendants should remain visible as superseded records. That would preserve the governance rule that a rollback does not silently replace the public manifest.
4. Loading-policy and loaded-dtype rehearsal
This probe used:
HuggingFaceTB/SmolLM2-135M-Instruct;- Tesla T4;
- torch
2.11.0+cu128;
- Transformers
4.57.6;
- PEFT
0.19.1;
- bitsandbytes
0.50.0;
- NF4 with double quantization and FP16 compute;
- a synthetic rank-8 LoRA over
q_proj and v_proj;
- 120 Adapter tensors and 60 resolved targets.
One correction is important: the saved native PEFT artifact contained FP32 Adapter tensors, despite the notebook introduction initially describing it as FP16.
Three fresh loading lanes were compared:
| Lane |
Actual loaded Adapter dtype |
PeftModel.from_pretrained, default autocast |
FP32 for all 120 tensors |
PeftModel.from_pretrained, autocast_adapter_dtype=False |
FP16 for all 120 tensors |
Transformers-native model.load_adapter() |
FP16 for all 120 tensors |
Observed results:
- Adapter-disabled/base logits were exactly identical across all lanes.
- PEFT no-autocast and Transformers-native loading matched exactly on the fixture.
- PEFT default-autocast versus the FP16 lanes produced:
- enabled-logit maximum absolute difference:
0.16796875;
- enabled-logit mean absolute difference:
0.0316916965;
- Adapter-effect cosine:
0.9996790111;
- Adapter-effect residual/reference:
0.0311880984.
The correct interpretation is therefore not that Transformers-native loading is inherently incompatible with PEFT-native loading.
The narrower result is:
The same artifact and base preparation can execute differently if the loading policy materializes Adapter parameters in a different dtype.
PEFT documents this behavior in the PeftModel API and troubleshooting guide.
For a promotion gate, it may therefore be useful to record or verify:
- loading/injection lane and relevant arguments;
- actual loaded Adapter dtype, not only saved-file dtype;
- resolved target names/counts;
- round-trip reference behavior;
- Adapter-enabled versus Adapter-disabled behavior.
Again, this is a deliberately visible synthetic fixture. It says nothing about the task-level importance of the difference for Qwen2.5-7B.
Candidate promotion, rollback cadence, and human-side gates
Candidate versus canonical
At current main, aggregation produces the new Adapter and outer velocity, registers the artifacts, and advances the current state in one path.
That is a sensible implementation for the current controlled system, but for the proposed Campaign it may be useful to name an intermediate state:
aggregate produced
→ candidate recorded
→ candidate verified
→ canonical pointer advanced
A candidate record could contain:
- parent canonical checkpoint;
- accepted delta/result IDs;
- aggregate Adapter/config hash;
- matching outer-state hash;
- structural validation report;
- fresh-process runtime identity;
- parity and activation-canary result;
- promotion/quarantine decision.
Promotion should advance Adapter and matching outer state atomically. If one is restored or promoted without the other, continued optimization can follow a state that never actually existed as a coherent checkpoint.
Verification cadence
I do not think the current evidence is enough to choose a numeric cadence.
A conditional rule seems safer:
Can the full fresh-process gate run for every aggregate
at a proportional cost?
├─ yes
│ └─ verify every aggregate before issuing descendant work
└─ no
└─ can a finite provisional depth K be declared and rehearsed?
├─ yes
│ └─ descendants remain explicitly provisional;
│ if the milestone gate fails, supersede the branch
└─ no or unknown
└─ do not issue descendant work
from an unverified aggregate
K should come from actual Campaign telemetry, not from a single Colab runtime:
- aggregate arrival rate;
- cold/warm reconstruction time;
- GPU and storage availability;
- promotion/quarantine queue age;
- human reviewer capacity;
- maximum tolerable rollback depth.
Unknown capacity should take the conservative branch.
Role acceptance and human capacity
The RFC says that a second maintainer and evaluation owner must accept responsibility before the RFC can move to Accepted.
The generic proposal validator already requires at least two distinct maintainer entries, a public decision log, conflict disclosure, and a nonempty rollback owner. That is a useful structural gate: proposal validation.
A Campaign-specific acceptance record could add the time-dependent part without requiring a large schema redesign:
role
+ stable public handle
+ accepted Campaign/amendment hash
+ responsibility horizon
+ backup or rehearsed handoff path
+ finite WIP/decision-age bounds
+ expiry/resignation/replacement path
Role overlap may be unavoidable in a small project, but three role names held by one person still represent one person’s capacity. The same person also cannot provide their own independent backup.
A human-side decision tree could be:
Critical integrity or canonical-state uncertainty?
├─ yes
│ └─ hard pause immediately
└─ no
└─ role coverage and decision queues within declared bounds?
├─ yes
│ └─ continue
└─ no or unknown
└─ can in-flight work drain safely?
├─ yes
│ └─ stop new leases, drain, then soft pause
└─ no or unknown
└─ hard pause and quarantine unresolved candidates
This treats pause as normal load shedding rather than project failure.
Emergency pause and resume do not need symmetrical authorization:
- an emergency pause should be fast and may be initiated by one authorized Operator;
- resuming after a corrective rollback can reasonably require reconciliation plus acknowledgement by another named maintainer.
Evaluation lifecycle, the 256 → 1,024 rule, and GSM8K scope
Development evidence and terminal confirmation
The previous showcase already demonstrates why evaluation roles matter.
An earlier 1e-4 Adapter improved validation loss while normalized exact match on the development set fell from 105/128 to 95/128. That attempt was not accepted as success. The final attempt then preregistered:
- learning rate;
- training budget;
- generation configuration;
- primary metric;
- practical threshold;
- a new disjoint confirmatory holdout.
This supports keeping validation loss and perplexity as diagnostics rather than treating them as substitutes for task performance.
For the proposed Campaign:
D = development/extension set
C = terminal confirmatory set
D may be used at 128 and 256 to decide whether continuing is justified.
C should be opened only for the actual terminal release or quality-claim candidate.
Possible lifecycle:
before training
→ pin D and C identities
→ pin evaluator, prompt/template, decoding,
parser, normalization, timeout/error rules
→ run or commit baseline according to the chosen exposure policy
128
→ D or operational kill-switch checks
→ continue only to 256
256
→ evaluate S, Q, and H
├─ stop without a terminal quality claim
│ └─ publish D result; C may remain sealed
├─ choose 256 as terminal candidate
│ └─ evaluate once on C
└─ amend and extend toward at most 1,024
└─ keep C sealed until the later terminal candidate
A valid negative or mixed result on C still consumes C. Otherwise the Campaign could modify the model after seeing the result and reuse the same supposed confirmation set.
A narrowly defined operational retry may be reasonable only if it is auditable that:
- no semantic metric was exposed;
- evaluation did not complete;
- checkpoint, evaluator, inputs, and configuration remain identical.
If a valid C result is seen and then the model, training lineage, evaluator, prompt, parser, or decoding contract changes materially, I would call that a new attempt requiring a new sealed C.
Baseline exposure
The current RFC says to publish the frozen-base baseline before accepting training updates.
There are two defensible branches:
Option A: preserve the current RFC rule
→ publish the C baseline before training
→ treat that as one limited exposure
→ prohibit model/config/parser/threshold selection based on it
Option B: stronger terminal isolation
→ compute and hash the C baseline before training
→ keep its value sealed
→ publish baseline and candidate together at terminal evaluation
The important part is to choose and preregister one policy.
Repeatedly inspecting the same fixed holdout while selecting checkpoints or settings can adapt the process to the holdout itself. This is the broader issue studied in adaptive data analysis and holdout reuse. An ordinary optional-stopping rule does not automatically solve repeated adaptive model selection on the same finite examples.
128-step gate
I would treat 128 primarily as a kill-switch rather than a success declaration.
Possible checks:
- artifact/runtime integrity;
- valid-answer-rate collapse;
- obviously harmful task regression;
- unresolved state or lineage inconsistency;
- unsustainable evaluation/incident queue.
If evaluation capacity is very constrained, the 128 gate could be operational only, preserving task-set capacity for 256.
Passing 128 would mean only:
No stop condition was observed; continuing to 256 remains permitted.
256-step gates
Systems gate (S)
Possible conditions:
- latest candidate passed promotion;
- Adapter and outer state reconcile;
- no unresolved quarantine or rollback inconsistency;
- resource envelope is measured and published;
- required recovery/rollback rehearsal is complete.
Quality gate (Q)
Possible preregistered elements:
- a practical improvement threshold;
- valid-answer-rate or other harm limits;
- treatment of conflicting metrics;
- confidence interval used only for the strength of the public claim.
The practical threshold is a resource-allocation rule, not automatically a statistical-significance claim.
The previous showcase handled this distinction well: the practical +2 pp rule passed, while the paired bootstrap interval included zero, so statistical significance was not claimed.
Human/resource gate (H)
Possible conditions:
- Operator, evaluation owner, maintainer, and backup coverage remain accepted;
- promotion/evaluation/incident queues are within their declared limits;
- additional GPU/runtime/download/energy costs are understood;
- rollback exposure remains bounded;
- contributors accepting new work receive the amended contract.
An extension should not occur merely because the quality metric is positive if the systems or human gate is not ready.
GSM8K and proportional supplementary evaluation
GSM8K still seems reasonable as the first Campaign objective as long as the claim remains exactly as bounded as the RFC states:
improve GSM8K-style mathematical answer accuracy without reducing valid-answer rate.
I would not make a broad benchmark bundle a launch condition. That would increase GPU, prompt/parser, and evaluation-owner burden while expanding the objective beyond the RFC.
A proportional portfolio could be:
| Role |
Candidate evidence |
D |
Campaign-fresh GSM8K development/extension evidence |
C |
Campaign-fresh terminal GSM8K confirmation |
M |
Optional external matched audit |
P |
Optional perturbation-sensitivity diagnostic |
These should be reported as separate results, not merged into one weighted score.
Freshness terms
It may help to distinguish:
- Campaign-fresh: not previously used to select this Adapter, settings, or extension decision;
- corpus-external: not drawn from the original GSM8K corpus;
- model-temporally-fresh: created after the frozen model weights.
A new unused GSM8K subset can be Campaign-fresh without being corpus-external or model-temporally fresh.
GSM8K-Platinum may be useful as cleaner same-corpus evidence because it rechecks and filters original GSM8K items. It is not a new external corpus and should not be described as temporally fresh.
GSM1k is more useful as matched external evidence, but the currently public official evaluation repository exposes a limited set rather than making the entire 1,000-item collection freely available. It should therefore remain optional and narrowly described.
GSM-Symbolic is better treated as a perturbation diagnostic than as an independent broad benchmark. Multiple instances derive from shared templates and retain links to original GSM8K items, so template-level dependence should remain visible.
Decision tree:
Can D and C be evaluated reliably within human capacity?
├─ no or unknown
│ └─ do not add supplementary suites;
│ protect the core evaluation
└─ yes
└─ can one terminal-only external matched audit be sealed?
├─ no
│ └─ report external transfer as untested
└─ yes
└─ add M
└─ only add P if capacity still remains
This keeps the Campaign’s primary claim bounded while giving future readers a clearer view of where the improvement does and does not transfer.
Compact amendment map, source links, and non-claims
Compact amendment map
| Area |
Minimal adoptable rule |
| Candidate state |
Quorum aggregation creates an immutable candidate |
| Promotion |
Structural check + fresh-process parity + activation contrast |
| Canonical transition |
Adapter and matching outer state advance atomically |
| Failure |
Quarantine candidate; canonical state remains unchanged |
| Provisional work |
Allowed only under a finite declared depth and supersession rule |
| Recovery |
Preserve valid work when returning to the same trusted lineage |
| Corrective rollback |
Restore a trusted ancestor and fence abandoned-lineage work |
| Evaluation |
Bind exact checkpoint, evaluator identity, and evidence role |
| Holdout lifecycle |
D for development/extension; C once for terminal confirmation |
| 128 |
Kill-switch, not success declaration |
| 256 |
Independent systems, quality, and human/resource gates |
| Roles |
Campaign-bound acceptance and finite responsibility horizon |
| GSM8K |
Keep the bounded primary objective; supplementary suites remain proportional |
Main project sources
Relevant external references
Scope
None of the suggestions above resolves the RFC’s major external blockers:
- ordinary Volunteer-path support for the pinned 7B runtime;
- two independently administered Internet-host E2E runs;
- the appropriate minimum useful work-unit size;
- permissionless or adversarial safety;
- poisoning, Sybil, or Byzantine resistance;
- secure aggregation;
- universal PEFT compatibility;
- broad reasoning improvement;
- a production SLA.
The direct 7B Colab run is not ordinary-path evidence. The PEFT checks are bounded runtime fixtures, not compatibility proofs. The rollback rehearsal is a state-contract observation, not a vulnerability report. The evaluation structure is a way to limit adaptive reuse, not a claim of mathematically optimal thresholds or sample sizes.
The value I see is mainly organizational:
different contributors could validate one bounded piece at a time without needing one person to simultaneously provide GPU access, PEFT expertise, evaluation design, distributed-systems review, operational availability, and long-term governance responsibility.
That seems compatible with the project’s existing emphasis on bounded claims and auditable evidence, while leaving the formal launch blockers honestly visible.