Project UCTF — Paper 1 Complete Results
Measuring Semantic Redundancy in Multilingual Training Corpora
Paper 1 is complete..
Full PDF report and all code, data, and notebooks are available on GitHub:
What we measured
We tested whether cross-lingual semantic redundancy in multilingual AI training data is measurable, substantial, and encoder-dependent — and whether an embedding-based approach can detect it reliably.
Experiment overview
Version | Scale | Setup | Mean similarity
v0 | 20 pairs | Manual aligned, 4 languages, MiniLM | 0.9570
v1 | 1,500 pairs | OPUS-100 real corpus, MiniLM | 0.7474
v2 | 1,500 pairs | 4 encoders compared on GPU | 0.8820 (mE5)
v3a | 900 sentences | 3 noise types, mE5 | 0.9627 avg
v3b | 600 pairs | Nepali, Sinhala, Kyrgyz, mE5 | 0.9045 avg
v4 | 10,000 pairs | 5 languages, corpus scale, GPU | 0.8948
Total pairs analyzed: 13,320
Languages tested: English, Chinese, Japanese, Russian, Nepali, Sinhala, Kyrgyz
Hardware: NVIDIA RTX 3050 4GB (consumer laptop GPU)
Finding 1 — Text deduplication fails completely
Exact duplicate detection rate: 0.0%
Standard text-level deduplication finds zero cross-lingual redundancy by design. The same sentence in English and Chinese shares no character overlap. This confirms that semantic redundancy measurement requires embedding-based approaches — not text matching.
Finding 2 — mE5-small is the strongest encoder baseline
Encoder comparison at 1,500 pairs (v2):
Encoder | Eng-Chi | Eng-Jap | Eng-Rus | Overall | Neg. pairs | Std dev
MiniLM | 0.7948 | 0.6589 | 0.7886 | 0.7474 | 1 | 0.1878
mE5 | 0.8694 | 0.8630 | 0.9136 | 0.8820 | 0 | 0.0455
LaBSE | 0.7898 | 0.6149 | 0.7855 | 0.7301 | 0 | 0.1990
mpnet | 0.8249 | 0.7044 | 0.8176 | 0.7823 | 0 | 0.1733
mE5-small outperforms all tested encoders on three criteria simultaneously: highest mean similarity (0.8820), lowest standard deviation (0.0455), and zero negative similarity pairs. It also reduces the English-Japanese divergence gap from 0.136 (MiniLM) to 0.006 — a finding that directly addresses the low-resource language concern raised in earlier feedback.
Finding 3 — mE5 is robust to real-world noise (v3a)
Noise type | Mean sim to clean | Above 0.85 threshold
Word shuffle (30% swapped) | 0.9750 | 100.0%
Char noise (5% OCR errors) | 0.9643 | 100.0%
Code switching (20% foreign) | 0.9487 | 98.7%
mE5 maintains 94.9–97.5% similarity to clean text under three types of realistic noise. This suggests the encoder is viable for real training corpora where perfect translations cannot be assumed.
Finding 4 — Low resource languages match high resource (v3b + v4)
At corpus scale (2,000 pairs each):
Language | Type | Mean similarity | Above 0.85
Sinhala | Low resource | 0.9184 | 93.5%
Russian | High resource | 0.9092 | 91.3%
Nepali | Low resource | 0.9041 | 89.3%
Chinese | High resource | 0.8755 | 77.6%
Japanese | High resource | 0.8667 | 67.2%
Sinhala (low resource) outperforms Chinese and Japanese (high resource) at corpus scale. This is counterintuitive and we do not have a confident explanation yet. It is flagged as a priority investigation for Paper 2.
Finding 5 — Consumer hardware is sufficient
20,000 sentences encoded in 13.2 seconds
1,515 sentences per second
458 MB VRAM used out of 4,096 MB available
Hardware: NVIDIA RTX 3050 Laptop GPU
Corpus-scale multilingual semantic redundancy measurement is practical on consumer hardware. This is relevant for the UCTF democratisation goal.
Limitations
OPUS-100 is a parallel translation corpus — cleaner than real web-crawled training data. Only 7 languages tested — broader coverage needed. Noise simulation is controlled — real corpus noise is more varied. No manual audit of candidate pairs completed yet. Negative similarity cases require manual inspection.
Conclusion
Cross-lingual semantic redundancy is measurable, substantial, and encoder-dependent. mE5-small is a viable baseline for a UCTF compression layer — robust to noise, effective across language families including low-resource settings, and practical on consumer hardware.
The core UCTF hypothesis — that a machine-native compressed representation could reduce multilingual training data redundancy — is supported by these preliminary findings.
What comes next — Paper 2
Characterizing which information is universal vs language-specific.
Manual audit of candidate pairs.
SOSAR and LaBSE comparison at corpus scale.
Test on web-crawled corpora (FineWeb2, CulturaX).
Investigate the low-resource language outperformance finding.
Resources
Full PDF report: github.com/kesavanvjp0007/project-uctf/paper1_final
All code and notebooks: github.com/kesavanvjp0007/project-uctf/paper1_v0 through v4
Candidate pairs CSV: 116 semantic pairs with similarity scores
Threshold sweep CSV: precision at 7 thresholds
Encoder comparison CSV: 6,000 records across 4 encoders
Feedback on the low-resource outperformance finding and the noise robustness results especially welcome.
Special thanks to @John6666 for the research architecture guidance that shaped this entire program.