Skip to content

ML engineeringIndependent open-source work · AstroCLIP

Integrating a multimodal model with Hugging Face

Adapted AstroCLIP's weights and preprocessing for image and spectrum encoding in Transformers workflows.

Result

Standard Hugging Face loading, checked against original outputs.

The challenge

Reusing AstroCLIP in a Transformers workflow requires compatible weights, preprocessing and loading code. The conversion also needs to preserve the original model's behaviour.

My contribution

  • Adapted the AstroCLIP checkpoint and preprocessing for AutoModel and AutoProcessor. Published the package on Hugging Face.
  • Converted the weights to Safetensors, mapping all 446 source keys without skipping any.
  • Checked image and spectrum embeddings, similarity scores and contrastive loss against the original implementation on fixed inputs.

Results and validation

The package loads through AutoModel and AutoProcessor and reproduces the original outputs on fixed test inputs within relative tolerance 1e-4 and absolute tolerance 1e-5.

Technical details

Model integration and fixed-input validation using the AstroCLIP authors' model and checkpoint. Custom-code loading requires trust_remote_code=True.