https://arxiv.org/html/2501.15316v2

ToMoE: Unlocking Hidden Experts Inside Dense LLMs — Comprehensive Summary

1. The Paper in One Paragraph

ToMoE (published in Transactions on Machine Learning Research, January 2026) demonstrates that dense Large Language Models already contain latent “expert” sub-networks within their MLP layers. Rather than permanently cutting parameters away (pruning) or rebuilding the model, ToMoE learns a lightweight routing mechanism that uncovers these experts — without ever modifying the original weights. The result is a sparse Mixture-of-Experts model that activates only ~50% of parameters per token yet consistently outperforms every competing pruning and MoE-construction method across six models (Phi-2, LLaMA-2 7B/13B, LLaMA-3 8B, Qwen-2.5 7B/14B), using just 0.02 billion training tokens and no fine-tuning of model weights.

2. The Core Phenomenon — Explained Simply

“Latent expert structures already exist inside dense pretrained LLMs and can be uncovered without altering the original weights, eliminating the need for extensive fine-tuning.”

The Swiss Army Knife Analogy

Think of a dense LLM’s MLP layer as a Swiss Army knife with 4,096 tools crammed into one handle. Every time you need to cut paper, the knife opens all 4,096 tools simultaneously — absurdly wasteful. What you actually need is just the scissors for paper, the screwdriver for screws, the bottle opener for bottles.

Here’s the key insight: those specialized tools already exist inside the handle. Nobody manufactured them separately. They were always there, overlapping, latent. You just need a mechanism to say: “For this task, pull out only tools #3, #17, #42, and #200.”

ToMoE builds that mechanism. It learns a tiny router (a single linear layer) that looks at each incoming token and says: “This token needs expert 3.” Expert 3 is not a new network — it is the same original weight matrix, but with a binary mask selecting only certain columns. Different tokens get different masks. The union of all masks covers nearly the entire original model.

Why This Is Technically Non-Obvious

A skeptic might object: “Isn’t this just pruning with extra steps?” The critical distinction:

Static Pruning ToMoE’s Dynamic Routing
Decision timing Once, before deployment Per-token, at every forward pass
Parameters removed? Yes, permanently No — all weights stay; only activation paths change
Total capacity Reduced permanently Union of all experts ≈ full dense model
Per-token cost Fixed (smaller) Fixed (same for every token, but ~50% of dense)
Reversibility Irreversible Remove routing modules → recover exact dense model

The paper states this directly: “Our findings reveal that MoE inherently exists within dense models and can be uncovered without updating model weights (continue pretraining).” The word “inherently” is doing heavy lifting — it claims the modular structure was learned implicitly during pretraining, not imposed after the fact.

The Technical Mechanism in Plain Terms

Inside every transformer layer, the MLP block computes:

Input → Project Up → Gate → Element-wise Multiply → Project Down → Output

ToMoE inserts a binary selection matrix (a vector of 0s and 1s) between these steps. For expert i, only the positions marked with 1 are active. The selection matrix is generated by:

  1. A router (one linear layer) looks at the token’s representation and outputs a score for each of N experts.
  2. The highest-scoring expert is selected (top-1 routing).
  3. That expert’s pre-learned binary mask is applied to the MLP’s weight matrices.

The mask is learned during a short training phase (10,000 iterations), but the original MLP weights never change. Only the router, small projection layers, and a hypernetwork are trained.

3. Expert Construction Strategy — Step by Step

3.1 Two Different Treatments for Two Different Components

Component Strategy Why
MLP layers Convert to N experts with top-1 routing along the intermediate dimension MLP has no pairwise token interaction → per-token masks are safe
MHA (attention) layers Static pruning on Q/K + dynamic top-K on V/O along head dimension Q·K dot-products require matching dimensions across positions → Q/K must be static

3.2 How MLP Experts Are Built

The original MLP has three weight matrices: $W_G$ (gate), $W_U$ (up), $W_D$ (down). ToMoE does not create new matrices. Instead, for expert $i$:

  • A binary diagonal matrix $S_i$ (containing 0s and 1s) selects which columns of $W_G$ and $W_U$ are active, and which rows of $W_D$ are active.
  • The computation becomes: apply the same MLP formula, but only through the selected positions.

The binary vector $s_i$ (the diagonal of $S_i$) is generated from a learned pipeline:

  • Router outputs a one-hot vector → selects expert embedding → projection maps it to the MLP dimension → Gumbel-Sigmoid rounds to binary.

3.3 How MHA Is Handled

  • Q and K: Same pruning mask $S_0$ for all tokens (static). This is mathematically necessary — the paper proves (Appendix F) that if token $a$ keeps dimensions {1,3,5} and token $b$ keeps {2,3,4}, their dot-product overlaps only on dimension 3, wasting capacity.
  • V and O: Per-token mask $S_t$ (dynamic top-K). These don’t participate in pairwise comparisons, so per-token variation is safe.
  • All heads share the same mask → uniform head dimension → parallel processing preserved.
  • RoPE compatibility: the mask respects RoPE’s sub-space structure by duplicating the first half.

3.4 The Hypernetwork Glue

A small Bi-GRU network generates expert embeddings for all layers simultaneously: $E_{\text{all}} = \text{HN}(z)$, where $z$ is a fixed random input. This introduces cross-layer dependencies — the experts in layer 5 are informed by the experts in layer 20. The paper notes this “accelerates the learning process in practice.”

3.5 Post-Training Cleanup

After training:

  • The HyperNetwork and Proj$^{\text{MLP}}_D$ are removed entirely (their outputs are saved as fixed embeddings).
  • The remaining overhead is just: one Router per MLP layer, and two small projection modules per MHA layer.
  • For LLaMA-2 7B: total added parameters = 0.0184B = 0.27% of the model.

4. Validation for the Skeptical Reader

4.2 The Union-of-Experts Check

Figure 6a shows that the union of all expert masks closely tracks the full dense model capacity across all 32 layers. This means no parameter is permanently lost — it is merely assigned to a specific expert’s activation path. The regularization $R_U$ explicitly enforces this.

4.3 Expert Visualization Shows Structured Behavior

Table 7 (LLaMA-2 7B, last layer): different experts handle different tokens in a syntactically coherent way. The paper states: “Each expert aligns syntax rather than semantic meanings, resembling the observations in (Jiang et al., 2024).”

Table 14 (math inputs): “Expert 2 in MLP 16 is predominantly activated by numbers and mathematical notations.” This is not random routing — it’s meaningful functional specialization.

4.4 Ablations Isolate Each Component

Removing any single design choice causes measurable degradation:

  • Replacing KL-divergence with language-modeling loss: −5.4 points (at p=0.5)
  • Removing Union-of-Experts regularization: −4.5 points (at p=0.4)
  • Switching head-dimension pruning to head pruning: −11 points
  • Removing global expert embeddings: −3 points

4.5 Training Cost Is Modest

Figure 6b: ToMoE’s training cost is comparable to DISP-LLM (a standard pruning method) and far below LLM Surgeon. The entire conversion uses 10,000 iterations on 1–4 A100 GPUs.

5. Impact & Significance (Expanded)

5.1 Reframing Model Compression: From Subtraction to Revelation

The dominant paradigm in LLM compression has been subtractive: start with a dense model, remove something (parameters, precision, layers), and then pay a performance debt that must be repaid through expensive fine-tuning. LLM-Pruner removes 50% of parameters and sees perplexity jump from 5.12 to 31.05 — a catastrophic 6× degradation. Even the best structural pruning methods (DISP-LLM, LLM Surgeon) show substantial gaps.

ToMoE inverts this paradigm entirely. It is revelatory rather than subtractive. The core insight — that functional experts already exist as latent structures within the dense weight matrix — shifts the compression question from “What can we afford to lose?” to “What is already there, and how do we activate it selectively?” No parameter is destroyed. No knowledge is erased. The model’s total capacity is preserved in aggregate; only the per-token activation is reduced.

This is not merely an engineering improvement. It represents a conceptual shift in how we think about over-parameterized networks: they are not monolithic blocks to be carved down, but implicit mixtures waiting to be decomposed.

5.2 Eliminating the Fine-Tuning Bottleneck

This is arguably the most practically consequential aspect. Prior dense-to-MoE methods require continued pretraining:

  • LLaMA-MoE: 1.2B tokens of fine-tuning
  • LLaMA-MoE-v2: 7B tokens of fine-tuning
  • G-MoEfication: requires retraining

ToMoE requires only 0.02B tokens — 350× fewer than LLaMA-MoE-v2 — and trains only small routing/projection modules (0.27% additional parameters). The original weights are provably unchanged.

Three downstream consequences:

  1. Democratization. Converting a 7B model costs 2 A100s for 10K iterations — roughly a few hours and a few hundred dollars. Labs without massive compute budgets can now produce competitive MoE models. Compare this to the weeks of GPU time needed for LLaMA-MoE-v2’s 7B-token fine-tuning.

  2. Safety preservation. Since original weights are frozen, the model’s alignment, safety training, and factual knowledge are provably unchanged. There is zero risk of catastrophic forgetting or alignment drift. For safety-critical deployments (medical, legal, aligned assistants), this guarantee is invaluable.

  3. Reversibility. Remove the routing modules, and you recover the exact original dense model. The conversion is non-destructive.

5.3 A Fixed Budget That Works in Production

Dynamic pruning methods like D-LLM skip layers adaptively, but the number of active parameters varies per token. This creates serious engineering problems:

  • Variable-length mini-batches
  • Unpredictable memory access patterns
  • Difficulty with KV-cache management during prefilling
  • Inability to use standard MoE serving infrastructure

ToMoE’s top-1 routing for MLP and fixed top-K for MHA guarantee the same compute budget for every token. The paper explicitly states: “The converted model maintains consistent computational costs for all inputs.” This makes it compatible with standard MoE serving infrastructure (vLLM, TensorRT-LLM MoE kernels). This is the difference between a research curiosity and a deployable system.

5.4 Bridging Two Research Communities

The paper sits at the intersection of pruning and MoE research, which have historically developed in parallel. The key conceptual bridge: “Conditional computation in MoE aligns closely with dynamic pruning: both make pruning decisions given input features.”

By showing that dynamic structural pruning is MoE construction (the routing mechanism learned for pruning directly serves as the MoE gate), ToMoE unifies the two fields. The regularizations (parameter budget, union-of-experts, load balancing) borrow from MoE literature; the differentiable discrete operations and structural constraints borrow from pruning literature.

This cross-pollination opens new research directions:

  • Can pruning-aware training produce models that are born as MoE?
  • Can MoE routing insights improve pruning criteria?
  • Is latent modularity a universal property of over-parameterized networks?

5.5 Evidence for Latent Modularity in Neural Networks

The deepest scientific contribution may be the empirical demonstration that over-parameterized dense networks contain meaningful modular structure. The MLP layers were trained as monolithic blocks, yet ToMoE shows that binary masks can carve them into functionally distinct experts that route tokens by syntactic role (Table 7) and, for math inputs, by semantic content (Table 14).

This resonates with broader findings in interpretability (superposition hypothesis, sparse autoencoders) and suggests that dense training implicitly learns a mixture structure that is simply never exploited at inference time. ToMoE provides a practical mechanism to exploit it.

5.6 Consistency Across Scales and Architectures

The method works across:

  • Architectures: Standard MHA (Phi-2), GQA (LLaMA-3), various attention mechanisms
  • Scales: 2.7B (Phi-2) through 14B (Qwen-2.5)
  • Training regimes: Different pretraining data and objectives

The paper acknowledges the gap narrows slightly at 13B/14B (“the performance gap between our method and other approaches is smaller”), which is expected as larger models have less redundancy. But the consistent improvement across all scales demonstrates the phenomenon is general, not model-specific.

5.7 Practical Efficiency

The additional parameter overhead is negligible:

  • LLaMA-2 7B: 0.0184B additional parameters = 0.27% of total
  • After training, HyperNetwork and Proj$^{\text{MLP}}_D$ are removed
  • Inference throughput at batch 1536: 2919 tok/s vs. 1858 tok/s dense = 57% speedup
  • Training cost comparable to standard structural pruning methods

6. Main Contributions (Expanded)

Contribution 1: Dense-to-MoE Conversion Through Dynamic Pruning

What it is: ToMoE is the first method to convert a dense decoder-only LLM into a functional MoE model by treating the conversion as a differentiable dynamic structural pruning problem. Specifically:

  • For MLP layers: the intermediate dimension $d_{\text{mid}}$ is partitioned into $N$ overlapping binary masks. A learned router assigns each token to exactly one expert (top-1). The expert is not a new set of weights — it is the original $W_U$, $W_G$, $W_D$ matrices multiplied by a selection matrix $S_i$.

  • For MHA layers: Query and Key are statically pruned along the head dimension (same mask for all tokens, preserving attention consistency), while Value and Output receive dynamic top-K routing (per-token masks).

Why it matters: Prior MoE-from-dense methods (LLaMA-MoE, G-MoEfication, CMoE) follow a two-stage pipeline: first construct experts (often by splitting or clustering neurons), then train a router separately. The paper explicitly criticizes this: “Previous methods constructing MoE from the dense model separate the expert construction and router training into two distinct stages, often leading to sub-optimal performance.”

ToMoE’s single-stage formulation means the expert masks and the router co-evolve under the same gradient signal, producing tighter integration. The empirical result is dramatic: at 50% active parameters on LLaMA-2 7B, ToMoE scores 56.07 average zero-shot accuracy vs. LLaMA-MoE’s 42.31 (with fine-tuning) — a 13.76-point gap, achieved without any weight updates.

Contribution 2: Joint Optimization of Routing and Expert Configuration

What it is: The training objective simultaneously optimizes four sets of parameters — the hypernetwork, the router, the MHA projections, and the MLP projections — under a composite loss with four terms:

  1. KL-divergence distillation (preserve dense model behavior)
  2. Parameter budget regularization (control active parameter count)
  3. Union-of-experts regularization (ensure full coverage)
  4. Load balancing regularization (prevent routing collapse)

Why it matters: The paper states: “Our approach leverages differentiable operations to enable efficient and flexible MoE constructions.” The key innovation is that expert construction is not a separate preprocessing step — it is embedded within the optimization. The masks and routing decisions are learned together, avoiding the sub-optimality of sequential approaches.

The interaction between regularizations is carefully designed: “The combination of Eq. 8 and Eq. 9 creates an interesting phenomenon where they encourage uniform allocation of width among experts.” This emergent uniformity (visible in Fig. 8) means all experts end up with similar sizes, simplifying inference.

Contribution 3: No Weight Updates — Self-Knowledge Distillation

What it is: The original model’s weights are completely frozen throughout training. The distillation signal comes from comparing the logits of the original model (teacher) with the logits of the same model equipped with ToMoE modules (student). The paper provides pseudo-code (Listing 1):

  1. Disable ToMoE modules → forward pass → teacher logits
  2. Enable ToMoE modules → forward pass → student logits
  3. Compute KL divergence

Why it matters: The paper explicitly notes: “This approach does not introduce overheads in terms of GPU memory.” Because both forward passes use the same weight tensors, no separate teacher model needs to be loaded. This is not merely a convenience — it provides a formal guarantee that the conversion cannot degrade the model’s learned representations. The KL divergence can only decrease or stay the same; it cannot introduce new errors into the weight space.

Contribution 4: Consistent Empirical Improvements Across Models and Scales

What it is: The paper evaluates ToMoE on six models across three task families (language modeling, zero-shot reasoning, few-shot benchmarks) against three categories of baselines (structural pruning, semi-structured pruning, MoE construction). ToMoE achieves the best or second-best result in every setting.

Why it matters: The paper’s own summary: “Even without fine-tuning the model weights, ToMoE consistently outperforms state-of-the-art pruning and MoE techniques across Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5 models.” A single-model result could be a fluke. Consistent improvements across architectures (GQA, MHA), scales (2.7B to 14B), and evaluation protocols demonstrate that the underlying phenomenon is general.

The expert count experiments provide additional validation: N=16 improves over N=8, but N=24 provides no further gain (“a too-large number of experts burdens the learning process”). This non-monotonic behavior is consistent with genuine expert specialization rather than arbitrary parameter partitioning.

Contribution 5: Comprehensive Analysis and Interpretability

What it is: Beyond benchmark numbers, the paper provides:

  • Training dynamics curves for all four loss components (Fig. 3)
  • Layer-wise width allocation visualizations showing highly non-uniform distributions (Figs. 5, 7)
  • Expert similarity heatmaps (Fig. 9)
  • Token-level routing visualizations revealing syntactic alignment (Tables 7, 11) and semantic specialization for math (Table 14)
  • Expert size distributions showing convergence to uniform widths (Fig. 8)
  • Inference throughput measurements (Table 15)

Why it matters: The paper states: “We hope these analyses provide valuable insights and guidance for future research in this area.” The visualizations transform ToMoE from a “method paper” into a “phenomenon paper.” The observation that “the first layer exhibits a more diverse token distribution, while subsequent layers prefer to assign continuous tokens to the same expert” provides concrete guidance for future MoE design.

Contribution 6: Practical Efficiency and Minimal Overhead

What it is: After training, removable modules are discarded. The remaining overhead:

  • One Router (Linear(d, N)) per MLP layer
  • Proj$^{\text{MHA}}_E$ and Proj$^{\text{MHA}}_D$ per MHA layer
  • Total for LLaMA-2 7B: 0.27% additional parameters

Why it matters: Many MoE methods introduce substantial parameter overhead (duplicating FFN weights for each expert). ToMoE’s experts share the same weight matrices — they differ only in which columns are activated. The total parameter count does not increase; only the routing overhead is added. The method also supports “pseudo-MoE” conversion (Eq. 17) for cases where the active parameter ratio is large, enabling more efficient training.

8. Honest Limitations

For intellectual honesty, the paper has real constraints:

  • Cannot exceed the teacher: Self-KD constrains outputs close to the dense model. No capability gain is possible.
  • Memory footprint unclear: All expert parameters must reside in memory even though only a subset activates per token. The paper does not report memory comparisons.
  • Scalability unverified: Tested up to 14B / 32 layers. Whether the Bi-GRU hypernetwork scales to 70B+ (80 layers) is unknown.
  • Reasoning tasks untested: No evaluation on GSM8K, MATH, HumanEval, or MBPP.
  • Top-1 routing limitation: Only one expert per token in MLP → less expressive than top-2 MoE (e.g., Mixtral).
  • Training still required: 10,000 iterations on 1–4 A100s. Not zero-cost, though far cheaper than alternatives.

9. Final Assessment

ToMoE makes a compelling, well-validated case that dense LLMs harbor latent expert structure that can be surfaced through learned dynamic routing — without touching a single original weight. The method is technically elegant (unifying pruning and MoE construction in a single differentiable framework), practically efficient (0.27% parameter overhead, 10K training iterations, 1–4 GPUs), and empirically dominant across six models and dozens of benchmarks.

Its most profound implication is conceptual: compression need not be destructive. The experts were always there; we just needed to learn how to ask for them.