For now, I tried partially testing it in Colab:
I think the direct answer to the broader question is yes: architectural directions outside the Transformer/LLM ecosystem still deserve serious investigation.
SEMA is particularly interesting because it does not merely replace one neural component with another. It combines explicit content-addressed memory, vector-symbolic similarity, instance-based transfer, weighted deduction, exact computation, and inspectable traces in one system.
Since the thread already contains a useful discussion of possible hybrid architectures, I focused on a narrower question:
What would help distinguish the capabilities already demonstrated by SEMA from the boundaries that still need to be measured?
I ran two small diagnostic passes against the pinned v0.2.4 release, at commit d14e55c, using Node 22.16.0 on a CPU-only Colab runtime.
The documented demo reproduced successfully, as did the selected upstream regression tests I ran. The tests below are not a general benchmark, and they are not intended as a verdict on the architecture. They only probe the local boundary around the current painter example.
My main result was:
The example clearly demonstrates structural answer-slot transfer followed by graph continuation. In this fixture, however, predicate-sensitive relation abstraction was not yet separated from those two capabilities.
That distinction seems useful because the documented result combines several operations in one successful answer:
- recognize a structurally similar stored case;
- identify an answer-like span;
- ground that span to
Pablo Picasso;
- follow the explicit memory edge from Picasso to another fact;
- compose and articulate the final answer.
The full pipeline works on the positive example. The interesting next step is to determine exactly which of these operations depends on the learned relation, which depends on positional or structural similarity, and which depends on later graph traversal.
What reproduced
With the documented deposits:
The Mona Lisa was painted by Leonardo da Vinci. → Leonardo da Vinci
The Starry Night was painted by Vincent van Gogh. → Vincent van Gogh
The Night Watch was painted by Rembrandt van Rijn. → Rembrandt van Rijn
Pablo Picasso → Pablo Picasso co-founded the Cubist movement
the unseen query:
The Weeping Woman was painted by Pablo Picasso.
produced:
Pablo Picasso co-founded the Cubist movement
This matches the documented demonstration.
The strongest positive result from the diagnostic pass was not only that the answer appeared, but that the two-stage dependency behaved causally when the stored evidence was changed:
| Intervention |
Observed result |
| Painter examples + Picasso continuation present |
Full Picasso continuation |
| Picasso continuation removed |
Pablo Picasso only |
| Painter examples removed |
The sentence query became silent |
Painter examples removed, but Pablo Picasso queried directly |
Atomic continuation still worked |
| Picasso continuation replaced |
The answer followed the replacement |
This gives reasonably direct evidence for the following path in this fixture:
sentence
→ entity/slot grounding
→ Pablo Picasso
→ explicit continuation edge
→ final fact
That is a useful result. It shows more than memorizing the complete final sentence, and it is compatible with the architecture described in HOW_IT_WORKS.md.
Where a small contrast set helped
I then changed only the relation wording while preserving the same sentence shape and person:
The Weeping Woman was restored by Pablo Picasso.
The Weeping Woman was inspired by Pablo Picasso.
The Weeping Woman was photographed by Pablo Picasso.
The Weeping Woman was painted near Pablo Picasso.
The Weeping Woman was not painted by Pablo Picasso.
In this fixture, these variants still grounded Picasso and usually continued to the stored Cubist fact.
When I removed the Picasso continuation and tested only the slot extraction behavior, the following all returned Pablo Picasso:
The Weeping Woman was painted by Pablo Picasso.
The Weeping Woman was restored by Pablo Picasso.
The Weeping Woman was not painted by Pablo Picasso.
This does not establish that SEMA cannot represent or learn predicate distinctions under another curriculum, representation, or configuration.
It does suggest that the current headline example, by itself, does not distinguish among several possible explanations:
- learning the relation
painted by;
- learning a broader
verb + by + person frame;
- identifying the differing suffix/answer span;
- transferring a structurally aligned slot from similar examples;
- applying one of several redundant grounding mechanisms that happen to agree on the positive example.
A few carefully selected contrasts may therefore be more informative than adding more same-template positive examples.
A practical minimal set could be:
X was painted by Y.
X was restored by Y.
X was not painted by Y.
Y painted X.
X was restored by A and painted by B.
Who painted X?
These separate at least four capabilities:
- structural slot transfer;
- predicate sensitivity;
- argument-role sensitivity;
- question/declarative transformation.
This is close to the motivation behind Contrast Sets and CheckList: rather than relying only on aggregate success, make small meaning-preserving or meaning-changing edits and inspect the local decision boundary.
Suggested default route
The lowest-cost, highest-information route seems to be:
- Pin one release, seed, fixture, deposit order, and runtime.
- Keep the existing positive example.
- Add 10–20 local contrast cases.
- Record the answer, provenance, and compact rationale for each case.
- Separate grounding, atomic continuation, and full composition.
- Delete or replace the stated supporting deposit and replay the query.
- Only then use narrow mechanism ablations where attribution is still ambiguous.
A public fixture could be very small. JSONL or CSV would be enough:
id
capability
deposits
query
expected_class
expected_relation_or_role
actual_answer
provenance
compact_rationale
version
commit
runtime
seed
deposit_order
store_or_fixture_checksum
Useful expected_class values might be:
exact_answer
acceptable_answer
justified_abstention
unsupported_answer
It may also help to score the relation or role separately from the final text. A system can sometimes reach the correct entity through the wrong relation, and final-answer matching alone would hide that distinction.
The repository already contains detailed regression fixtures, including a particularly useful test where a weakly grounded comparison previously returned an unrelated haiku continuation and the corrected behavior is “honest silence.” That test is a good precedent for this style of boundary case:
test/50-cast-analog-consensus-floor.test.mjs
So this may not require creating a new evaluation culture. It may mainly require organizing some existing and new regression cases by capability and expected outcome.
Detailed observations from the local contrast probes
1. Number of worked examples
I tested the positive painter query with one, two, and three painter examples.
| Number of painter examples |
Observed output |
| 1 |
Picasso continuation |
| 2 |
Leonardo fragment + Picasso continuation |
| 3 |
Picasso continuation |
In this small fixture, the headline answer was therefore already obtainable from one worked example.
That does not show that three examples are generally unnecessary. It means only that the final answer alone cannot establish that this particular output depended on induction over all three examples.
The two-example condition was also informative because it produced a fused answer rather than a monotonic improvement. That suggests example count and the set of competing attention points may deserve their own stability test.
2. Active voice and argument roles
The passive input:
The Weeping Woman was painted by Pablo Picasso.
worked as documented.
The active-voice version:
Pablo Picasso painted The Weeping Woman.
produced:
Pablo Picasso painted by Leonardo da Vinci.
The trace indicated that this involved a different grounding/substitution path and a connector miss.
I would treat this as a useful argument-role boundary case, not as a broad conclusion about active voice. A compact role test could hold the entities and relation constant while varying only:
X was painted by Y.
Y painted X.
A told B that Y painted X.
X was restored by A and painted by Y.
3. Question form
The declarative query:
The Weeping Woman was painted by Pablo Picasso.
returned the Picasso continuation.
The question:
Who painted The Weeping Woman?
was silent in this fixture.
That may be entirely consistent with the supplied training format, which teaches context-to-continuation mappings rather than an explicit declarative-to-interrogative transformation.
It may therefore be useful to treat these as different capability levels:
declarative completion
relation-sensitive extraction
question transformation
open-ended question answering
4. Unknown entity
For:
The Weeping Woman was painted by Alice Example.
the system returned:
Alice Example
rather than silence.
This appears consistent with successful answer-slot extraction followed by the absence of a known continuation.
For evaluation purposes, it would be useful to distinguish:
- correct slot extraction;
- successful stored-fact continuation;
- complete task answer;
- justified abstention.
Otherwise, a partial intermediate result may be counted either as success or failure depending on the evaluator’s unstated expectation.
5. Negation and fusion
For:
The Weeping Woman was not painted by Pablo Picasso.
one standard condition returned:
Leonardo da VinciPablo Picasso co-founded the Cubist movement
The trace separated this into approximately:
CAST grounds the Picasso continuation
+
another attention point contributes a Leonardo fragment
+
fusion combines them
+
no connector is found
+
bare concatenation is emitted
I then tested a diagnostic build with only the downstream fusion step disabled.
The extra Leonardo fragment disappeared, but the positive Picasso continuation remained.
That suggests two separable boundaries:
- an upstream relation/negation grounding issue;
- a downstream fusion/articulation issue that can amplify the malformed result.
This is a good example of why the trace is valuable. It permits a malformed answer to be localized instead of treating the whole system as one opaque pass.
The fusion-disabled condition was only a diagnostic patch to a copied compiled build, not a supported configuration or proposed fix.
6. Redundant mechanisms
The headline positive answer could be produced through more than one route.
- With CAST disabled, extraction could ground Picasso and reasoning could continue.
- With extraction disabled, CAST could produce the route.
- With both removed, recall could still supply a related route in some conditions.
- Removing an individual mechanism therefore did not necessarily remove the headline capability.
This may be desirable redundancy, but it complicates attribution.
Ablation conclusions should therefore be phrased carefully:
“The output survived removal of mechanism A”
does not necessarily mean:
“Mechanism A contributes nothing.”
It may mean that another mechanism implements an overlapping capability.
Combined ablations are useful only after the behavioral boundary is clear; otherwise they can become a search through implementation combinations without a stable capability definition.
7. Explicit relation-contrast deposits
I also tried a small exploratory condition where painted, restored, and not painted examples were deposited with different intended continuations.
In that particular setup, the tested relation-specific expectations were not recovered.
This should not be read as proof that relation-specific learning is impossible in SEMA.
It only shows that ordinary context-to-continuation deposits in that small curriculum did not immediately disambiguate the relations. Other possibilities remain open:
- a different example curriculum;
- explicit relation nodes;
- a representation designed for predicate/argument structure;
- additional negative or contrastive evidence;
- a different mechanism boundary;
- a future dedicated interface for constraints or negative cases.
The relevant comparison concept here may be anti-unification, which studies how to extract a least-general common structure from examples.
That is not a claim that SEMA is “just anti-unification.” It is a useful baseline question:
What part of the demonstrated transfer requires SEMA’s vector-symbolic geometry, and what part could also be explained by ordinary common-structure extraction?
A capability ladder that may help organize future tests
The broad phrase “language understanding and reasoning” contains several separable capabilities.
A possible ladder is:
| Level |
Capability |
Minimal diagnostic |
| 0 |
Exact memory / atomic retrieval |
Query a stored key directly |
| 1 |
Structural answer-slot transfer |
New entity in a familiar frame |
| 2 |
Predicate sensitivity |
painted versus restored |
| 3 |
Argument-role sensitivity |
Passive versus active; two entities |
| 4 |
Negation, unknown, and conflict |
Positive, negative, unsupported, contradictory |
| 5 |
Multi-hop composition |
Ground an intermediate and continue |
| 6 |
Memory lifecycle |
Corpus growth, correction, retraction, stale knowledge |
The current painter fixture gives useful evidence for levels 0 and 1, and for part of level 5.
The local contrast probes suggest that levels 2–4 should be measured independently rather than inferred from the positive level-5 output.
This also prevents a system from being judged too harshly or too generously:
- failing question transformation does not erase successful structural transfer;
- successful entity extraction does not by itself establish predicate understanding;
- successful atomic retrieval does not establish composition;
- successful composition in one template does not establish role or negation handling.
Suggested decision tree
Can the pinned documented example be reproduced?
|
+-- No
| |
| +-- First fix the reproduction tuple:
| version, commit, runtime, seed, fixture,
| deposit order, configuration, expected trace
|
+-- Yes
|
+-- Run predicate, negation, and role contrasts
|
+-- Wrong entity or span
| |
| +-- Inspect recognition, CAST, extraction,
| halo evidence, and attention roots
|
+-- Correct entity, wrong continuation
| |
| +-- Inspect edge selection, recall,
| conflict resolution, and stored continuations
|
+-- Atomic stages succeed, combined answer fails
| |
| +-- Inspect multi-hop, fusion,
| connector selection, and articulation
|
+-- Result changes only with broader candidate search
| |
| +-- Investigate candidate-retrieval/index recall
|
+-- System is silent on nearly everything
| |
| +-- Measure answered-query risk together with coverage
|
+-- Local contrasts pass
|
+-- Move to held-out templates, longer hops,
unrelated facts, contradictions,
corpus growth, correction, and retraction
The default path would be to stop after the first few branches until each failure can be localized. A large benchmark becomes more useful after the expected behavior of these local cases is explicit.
How I would interpret provenance and auditability
SEMA’s trace system was one of the most useful parts of the experiment.
For future evaluation, I would separate four properties:
1. Execution consistency
Does the trace accurately record the mechanism and edges that actually ran?
2. Support sufficiency
Are the displayed deposits and derivation steps sufficient to support the answer?
3. Causal necessity
If a displayed supporting deposit is deleted or replaced, does the answer change in the predicted way?
4. Minimality
Does the rationale avoid presenting unrelated evidence as necessary support?
The evidence deletion and replacement probes gave encouraging results for causal sensitivity in the painter fixture.
This distinction may be helpful because “a trace exists” and “the trace is a faithful explanation” are not exactly the same claim.
A small replayable fixture is probably more informative here than a prose explanation alone:
original deposits
query
answer
trace
minus supporting deposit
same query
new answer
replacement supporting deposit
same query
replacement-following answer
Abstention and unsupported answers
The repository already treats “honest silence” as an important target, which seems appropriate for this architecture.
It may help to report several outcomes separately:
correct answer
justified abstention
excessive abstention
false grounding
retrieval omission
unsupported composition
near-match or exemplar echo
A low unsupported-answer rate can always be obtained by answering fewer queries, so abstention quality should be evaluated together with answer coverage.
The selective-prediction literature discusses this as a risk/coverage trade-off; a useful recent overview is On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing.
The methodology is applicable even though SEMA is not a neural selective-prediction model:
- vary a confidence or grounding threshold;
- measure how many queries receive answers;
- measure error among answered queries;
- separately count cases where the system remained silent despite sufficient stored evidence.
Later-stage comparisons, after the local boundary is clearer
These do not all need to be done now. They are possible next branches after the small contrast suite becomes stable.
VSA contribution
VSA/HDC is a family of representations and binding operations, not one uniform algorithm. Capacity, bundling, and approximate unbinding behavior differ among variants.
A useful comparison would be:
current SEMA representation
independent random byte vectors
character n-gram retrieval
edit-distance nearest case
exact graph-only retrieval
simple common-structure extraction
The question would not be whether VSA is “good” or “bad,” but:
Which tested capability disappears or degrades when the VSA-specific component is replaced?
A broad comparison of VSA families is available in A Comparison of Vector Symbolic Architectures.
Longer relational composition
Once local role and predicate contrasts work reliably, benchmarks such as:
- CLUTRR, for relation-chain generalization and distracting facts;
- ProofWriter, for proof depth and True/False/Unknown distinctions;
- COGS, for familiar components in unseen structural combinations;
could provide useful design ideas.
I would borrow their evaluation dimensions before attempting a full benchmark port.
Candidate retrieval and scaling
The exact graph can only verify candidates that the approximate index has surfaced.
At larger scale, useful measurements would therefore include:
candidate recall at k
final-answer agreement with a reference search
abstention agreement
trace agreement
query latency
index size
index build time
ANN-Benchmarks is useful for the general recall/speed methodology, although an end-to-end SEMA evaluation also needs to measure whether candidate differences change the final derivation.
Multilingual behavior
“Multilingual” may be easier to evaluate if divided into:
arbitrary UTF-8 storage
script independence
tokenizer-vocabulary independence
explicit cross-language aliases
parallel-pair transfer
shared non-text grounding
zero-link cross-language generalization
These are different capabilities.
Byte-addressed identity also raises a practical distinction between exact stored bytes and user-facing string equivalence. The W3C Character Model for the World Wide Web: String Matching and Searching gives useful background on Unicode normalization and equivalent-looking strings.
Memory lifecycle
A long-running explicit memory eventually needs policies for:
correction
retraction
contradictory sources
duplicate evidence
source independence
time-bounded facts
derived-fact invalidation
This is not specific to SEMA. It is a standard concern in case-based and explicit knowledge systems.
The classic retrieve/reuse/revise/retain cycle from case-based reasoning may provide useful vocabulary without replacing SEMA’s own terminology.
A compact proposed public probe
A very small first public suite could look like this:
| Group |
Example |
Expected distinction |
| Positive control |
X was painted by Y |
Structural transfer |
| Predicate contrast |
X was restored by Y |
Different relation |
| Negation |
X was not painted by Y |
Do not preserve positive entailment |
| Role reversal |
Y painted X |
Preserve argument roles |
| Two relations |
X was restored by A and painted by B |
Select the correct role |
| Unknown |
Known frame, unknown continuation |
Partial extraction or abstention |
| Atomic hop |
Query Y directly |
Verify continuation |
| Composition |
Frame query leading to Y |
Verify two-stage path |
| Evidence deletion |
Remove Y → fact |
Full answer should disappear |
| Evidence replacement |
Replace Y → fact |
Answer should follow replacement |
| Irrelevant additions |
Add unrelated deposits |
Positive result should remain stable |
For each test, recording a compact trace and expected outcome class would make the result useful to both human reviewers and automated regression tooling.
Overall impression
The experiment left me with a more positive and more specific view than either “it works” or “it fails.”
What seems demonstrated in the current painter fixture is:
- explicit atomic memory;
- structural case/slot transfer;
- continuation through an explicit stored relation;
- useful internal traces;
- causal sensitivity to supporting-deposit deletion and replacement;
- exact arithmetic through the separate ALU path.
What remains useful to isolate is:
- predicate-sensitive abstraction;
- argument-role preservation;
- negation and unknown handling;
- mechanism attribution when several routes are redundant;
- stability under competing examples and corpus growth;
- calibrated silence versus unsupported answers.
That seems like a productive position for an experimental architecture.
The components have substantial precedent individually, but the interesting research question is what capabilities emerge from this particular integration—and under what minimal conditions they stop emerging.
A small, versioned, trace-bearing contrast suite around the existing regression tests may be the most efficient next artifact for making that boundary visible to other researchers.