Enabling Edge Intelligence through Variational Autoencoder-Based Model Compression

Paper Info

Title	Enabling Edge Intelligence through Variational Autoencoder-Based Model Compression
Authors	Liang Cheng, Peiyuan Guan, Amir Taherkordi, Dapeng Lan
Conference	IEEE 11th International Conference on Edge Computing and Scalable Cloud (EdgeCom), 2025
Paper	DOI: 10.1109/EdgeCom66327.2025.00031
Code	GitHub: vae-model-compression-edgecom2025

---

Motivation

Deploying deep neural networks on resource-constrained edge devices (smartphones, IoT sensors, embedded systems) remains a fundamental challenge. Modern models have millions to billions of parameters — far exceeding the memory, storage, and computational budgets of typical edge hardware.

This problem is particularly critical for edge computing because:

1. Offloading to the cloud introduces latency — real-time applications (autonomous driving, AR, industrial sensing) can’t tolerate round-trip delays
2. Privacy concerns — sending raw data to the cloud violates data sovereignty requirements
3. Bandwidth is limited — constantly transmitting model updates over wireless networks is costly

Existing model compression techniques each have limitations:

Method	How it works	Key Limitation
Pruning	Remove redundant weights/neurons	Iterative retraining needed; hard upper limit on compression rate
Quantization	Reduce numerical precision (32-bit → 8-bit)	Beyond ~20×, accuracy drops sharply
Knowledge Distillation	Train a small “student” to mimic a large “teacher”	Requires access to strong teacher models; computationally expensive
Low-Rank Factorization	Decompose weight matrices	High computational cost; domain-specific

Common drawback of all above: they produce a static compressed model. There’s no mechanism for on-demand adaptation — you can’t temporarily decompress to higher accuracy when edge resources allow, or recompress for transmission.

---

Method

Core Idea: Compression as a Generative Problem

Instead of removing parameters (pruning) or reducing precision (quantization), we learn a compact latent representation of the parameter distribution using a Variational Autoencoder (VAE).

            ┌──────────┐                        ┌──────────┐
Parameters → │ Encoder  │ ─── latent vector z ──→ │ Decoder  │ → Reconstructed params
 (flattened) └──────────┘     (64-dim, tiny)     └──────────┘
   (2048-dim chunks)                              (2048-dim chunks)

Why this is powerful for edge computing:

• The latent vector is extremely small — ideal for storage and transmission
• The decoder can live on a server or on the edge device itself
• A server sends only the tiny latent vector; the edge device decodes it into a full-precision model on-demand
• This enables dynamic adaptation: compress more for transmission, decompress fully for inference

Pipeline

Step 1: Parameter Flattening & Chunking

Neural network parameters from all layers are flattened into a single 1D vector, then split into fixed-size chunks of 2048 dimensions. The last chunk is zero-padded if needed (padding length is recorded for reconstruction).

Layer 1 weights → flatten → [w₁, w₂, ..., wₙ] ──┐
Layer 1 biases  → flatten → [b₁, b₂, ..., bₘ]   │
Layer 2 weights → flatten → ...                  ├── concat → 1D vector → chunk into 2048-dim segments
...                                              │
Layer N ...                                     ─┘

This chunking mirrors edge device memory constraints — devices with limited RAM can process one chunk at a time.

Step 2: Data Augmentation via Noise Injection

Training a VAE requires multiple samples from the parameter distribution. Since training many high-performance models is expensive, we generate synthetic variations by injecting Gaussian noise (σ=0.01) into a single well-trained model’s parameters:

• 80 variations → training set
• 20 variations → validation set
• Original parameters → test set (100% ground truth)

Step 3: VAE Architecture

Component	Layers
Encoder	Linear(2048→1024) → Linear(1024→512) → Linear(512→512) → split into μ and log_var (latent_dim)
Decoder	Linear(latent→512) → Linear(512→512) → Linear(512→1024) → Linear(1024→2048)

The objective is the standard ELBO (Evidence Lower Bound):

$\text{ELBO} = \mathbb{E}_{q(z|x)}[\log p(x|z)] - \text{KL}(q(z|x) \parallel p(z))$

• First term: reconstruction loss (MSE between original and reconstructed parameters)
• Second term: KL divergence regularizer (keeps latent space well-structured)

Training uses the reparameterization trick for backpropagation through the stochastic sampling.

Step 4: Training Details

• Optimizer: Adam (lr=1e-3)
• Epochs: 500, with early stopping (patience=200)
• KL weight annealing: linearly increases from 0 to 1 over the first 100 epochs (prevents KL vanishing)
• A separate VAE is trained for each neural network architecture

---

Validation

Experimental Setup

Item	Details
Dataset	MNIST handwritten digits (60k train / 10k test)
Architectures	FNN, CNN, RNN, LSTM — four fundamental building blocks
VAE chunk size	2048
Latent dimension	64 (compression ratio ~32× per chunk)
Platform	Apple M3 Pro, 18 GB RAM, PyTorch

Model Architectures

Model	Layer Structure	Parameters
FNN	[784, 200, 100, 60, 30, 10]	185,300
CNN	3 conv layers + 2 FC	122,270
RNN	2-layer RNN (hidden=128) + FC	54,538
LSTM	2-layer LSTM (hidden=128) + FC	214,282

Key Results

Accuracy Comparison (MNIST Test Set)

Model	Original Accuracy	Reconstructed Accuracy	Accuracy Drop	Compression Ratio
FNN	98%	97%	~1%	~32×
CNN	98%	97%	~1%	~32×
RNN	90%	89%	~1%	~32×
LSTM	98%	98%	~0%	~32×

Overall: ~30× compression with <1% accuracy degradation — exceeding the typical practical limits of quantization (~20×).

Training Behavior

• No overfitting within 500 epochs across all architectures
• FNN converges fastest — despite having the second-largest parameter count. This suggests that parameter distribution and internal structure matter more than sheer size for VAE compression efficiency
• FNN’s fully-connected parameters have simpler, more regular patterns than convolutional or recurrent weights, making them easier for the VAE to learn
• VAE training time scales with input size — roughly proportional to the parameter count of the target model

---

Key Takeaways

Dimension	Finding
Main contribution	First to frame model compression as a generative VAE learning problem, enabling dynamic on-demand decompression
Compression rate	~30× with <1% accuracy loss — outperforms traditional quantization’s practical limit
Generality	Works across FNN, CNN, RNN, and LSTM — fundamental building blocks of modern deep learning
Edge advantage	Tiny latent vectors minimize storage and bandwidth; decoder can be hosted server-side or on-device
Dynamic flexibility	Unlike static pruning/quantization, VAE compression supports on-demand decompression and variable compression ratios
Limitation	Proof-of-concept on small MNIST models; scaling to large-scale architectures (Transformers, ViT) requires further work
Future direction	Hardware-aware compression for NPUs/TPUs, integration with federated learning (transmit latent vectors instead of full models), dynamic real-time compression adaptation

One-sentence summary: A VAE-based generative compression framework achieves ~30× model size reduction with <1% accuracy loss across four fundamental neural network architectures, enabling dynamic and efficient edge intelligence deployment.

---

BibTeX

@inproceedings{cheng2025enabling,
  title={Enabling Edge Intelligence through Variational Autoencoder-Based Model Compression},
  author={Cheng, Liang and Guan, Peiyuan and Taherkordi, Amir and Lan, Dapeng},
  booktitle={2025 IEEE 11th International Conference on Edge Computing and Scalable Cloud (EdgeCom)},
  pages={144--149},
  year={2025},
  doi={10.1109/EdgeCom66327.2025.00031}
}