EarlyLayers 2-9
Input-form separation. Expert selection differed by input form.
Expert selectionby decoder depth
Expert selection differs in layers 2 to 9. Expert lists become more similar for matching transcripts in layers 10 to 34.
TLDR
The router selects more of the same experts in deeper layers.
The router assigns each token to six experts. We compare the expert IDs across speech, image, and text.
In layers 2 to 9, the selected IDs differ by input form. Between layers 10 and 34, the lists become more similar. After layer 35, the router often selects similar lists for all three input forms.
Changing the early expert groups increases loss for every task. Similar expert lists do not prove that the hidden states have the same meaning.
Question
At which layers does expert selection align across speech, image, and text?
Inkling-Small has one decoder for all three input forms. It has no separate vision or audio Transformer.
The router assigns each token to six of 256 experts. An expert is a small processing block inside the decoder.
We provided the same sentence as speech, a rendered image, and text.
We recorded expert selection at each decoder layer. We then changed expert groups and measured task loss.
Expert selection identifies the active processing blocks. It does not measure similarity between internal representations.
Expert selection for one token
Select text, image, or audio. Then select one token. The chart lists the six selected experts at four layers.
LibriSpeech sample 1272-128104-0000
MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
8 shown from 28 text tokens
A left mark indicates a leading space.
12 shown from 136 image tokens
The experiment rendered this transcript at 640 by 320 pixels.
Evenly sampled across the image-token sequence
12 shown from 118 audio tokens
Sampled across 5.855 seconds.
Selected tokenText token 15, “CL” with leading space
Read from top to bottom. Each block is one selected expert. Block width represents routing weight.
L2early layer
E323.7%
E14121.3%
E16319.6%
E12012.8%
E20712.3%
E6010.3%
L21middle layer
E8619.2%
E19318.1%
E4516.9%
E13216.7%
E17115.5%
E5413.6%
L30middle layer
E10338.8%
E19127.0%
E10815.5%
E298.8%
E2095.3%
E2334.7%
L41late layer
E16325.7%
E5324.1%
E313.3%
E20113.1%
E1011.9%
E11711.9%
Interpretation
For this token, E3 has the highest weight at layer 2. E163 has the highest weight at layer 41. No expert ID repeats between consecutive shown layers.
The six saved weights add to 100% in each row. Color identifies the expert ID.
Convergence
Expert lists became more similar with depth.
We tested 18 transcripts. Expert selection differed by input form in layers 2 to 9.
In layers 10 to 34, matching transcripts produced more similar expert lists.
After layer 35, the same expert groups often received tokens from all three input forms.
This pattern is consistent with a transition from input-form processing to cross-modal content processing. Latent similarity requires activation-level measurements.
Test
We changed the experts at each stage.
Six transcripts identified the expert groups. Twelve different transcripts tested each change.
MiddleLayers 10-34
Matching transcripts. Expert lists were more similar for inputs with the same transcript.
LateLayers 35-41
Similar lists. Expert lists were often similar across input forms.
Route similarity alone does not establish causal importance. We blocked each group and measured the change.
We also swapped early groups between speech, image, and text.
The control condition changed other experts with similar overall use.
Early-group changes increased loss in all three tasks.
“Worse” marks a clear loss. “No clear change” includes small or uncertain differences.
| Task | Early | Middle | Late |
|---|---|---|---|
| Read speech | Worse | No clear change | No clear change |
| Read image text | Worse | No clear change | No clear change |
| Predict text | Worse | Worse | No clear change |
See exact effects and confidence intervals
Task
Earlyinput-form experts
Middlesame-transcript experts
Latecommon experts
ASR
+0.218[+0.105, +0.332]control +0.027
+0.014[-0.027, +0.051]control +0.027
-0.007[-0.022, +0.003]control -0.011
OCR
+0.354[+0.205, +0.524]control +0.072
+0.000[-0.000, +0.001]control +0.007
-0.000[-0.000, +0.000]control +0.000
Text
+0.107[+0.012, +0.212]control +0.065
+0.212[+0.104, +0.335]control -0.097
+0.013[-0.009, +0.035]control -0.008
Same-transcript retrieval accuracy also decreased.
This score measures whether two inputs contain the same words. Higher values indicate better retrieval. The control value did not decrease.
Original model72.2%
Early experts removed52.8%
Early experts swapped51.4%
Control change87.5%
In four generated examples, speech word error rose from 4.8% to 10.0%. Image-text character error rose from 0% to 4.7%.
Answer
Expert lists become more similar in the middle layers.
Expert selection differed by input form in layers 2 to 9. Blocking these groups increased loss for every task.
Expert lists for matching transcripts became more similar in layers 10 to 34. Changes in this range increased only text-prediction loss.
Expert lists were often similar across input forms after layer 35. Changes in this range had no clear effect.
These results measure convergence of expert selection across input forms. They do not measure convergence of internal representations.
A latent-alignment test requires expert activations for matching inputs at each layer.