---
tags:
- weight-space-learning
- neural-network-autoencoder
- autoencoder
- transformer
datasets:
- maximuspowers/muat-fourier-5
---

# Weight-Space Autoencoder (TRANSFORMER)

This model is a weight-space autoencoder trained on neural network activation weights/signatures.
It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).

## Model Description

- **Architecture**: Transformer encoder-decoder
- **Training Dataset**: maximuspowers/muat-fourier-5
- **Input Mode**: signature
- **Latent Dimension**: 256

## Tokenization

- **Chunk Size**: 1 weight values per token
- **Max Tokens**: 64
- **Metadata**: True

## Training Config

- **Loss Function**: contrastive
- **Optimizer**: adamw
- **Learning Rate**: 0.0001
- **Batch Size**: 32

## Performance Metrics (Test Set)

- **MSE**: 0.088629
- **MAE**: 0.186645
- **RMSE**: 0.297706
- **Cosine Similarity**: 0.9609
- **R² Score**: 0.9855