all-MiniLM-L6-v5-pair_score

This is a sentence-transformers model trained. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'siamy wrap',
    'siamy',
    'hair revival',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 2
• warmup_ratio: 0.1
• fp16: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0070	100	19.5113
0.0140	200	19.2906
0.0210	300	19.6854
0.0280	400	18.3384
0.0350	500	19.5149
0.0420	600	18.8415
0.0489	700	18.8726
0.0559	800	19.4663
0.0629	900	19.9079
0.0699	1000	18.8692
0.0769	1100	18.1987
0.0839	1200	17.8522
0.0909	1300	18.8869
0.0979	1400	18.5664
0.1049	1500	17.1952
0.1119	1600	18.9827
0.1189	1700	16.5597
0.1259	1800	16.4406
0.1328	1900	16.1171
0.1398	2000	16.2726
0.1468	2100	18.8375
0.1538	2200	17.7006
0.1608	2300	15.7486
0.1678	2400	17.8749
0.1748	2500	18.0333
0.1818	2600	15.5849
0.1888	2700	14.0833
0.1958	2800	12.0631
0.2028	2900	12.7098
0.2098	3000	11.1801
0.2168	3100	11.2842
0.2237	3200	13.884
0.2307	3300	14.7762
0.2377	3400	12.5363
0.2447	3500	11.2903
0.2517	3600	10.1627
0.2587	3700	11.1758
0.2657	3800	15.2709
0.2727	3900	12.5853
0.2797	4000	11.8216
0.2867	4100	11.4522
0.2937	4200	12.4326
0.3007	4300	12.6818
0.3076	4400	11.9052
0.3146	4500	12.0142
0.3216	4600	10.9577
0.3286	4700	11.4502
0.3356	4800	10.6586
0.3426	4900	11.4709
0.3496	5000	10.6245
0.3566	5100	10.305
0.3636	5200	10.3784
0.3706	5300	10.4824
0.3776	5400	9.861
0.3846	5500	10.9763
0.3916	5600	11.2106
0.3985	5700	13.8056
0.4055	5800	9.576
0.4125	5900	11.1156
0.4195	6000	11.8297
0.4265	6100	12.3616
0.4335	6200	10.0882
0.4405	6300	11.9186
0.4475	6400	11.0648
0.4545	6500	11.6143
0.4615	6600	10.476
0.4685	6700	10.6218
0.4755	6800	9.9786
0.4825	6900	10.167
0.4894	7000	10.6891
0.4964	7100	10.9375
0.5034	7200	10.0685
0.5104	7300	9.6487
0.5174	7400	9.4383
0.5244	7500	9.6336
0.5314	7600	10.4266
0.5384	7700	9.9414
0.5454	7800	9.426
0.5524	7900	9.834
0.5594	8000	9.8532
0.5664	8100	9.8928
0.5733	8200	9.5169
0.5803	8300	8.9786
0.5873	8400	9.0787
0.5943	8500	10.8244
0.6013	8600	9.581
0.6083	8700	9.7036
0.6153	8800	9.5971
0.6223	8900	9.547
0.6293	9000	9.0646
0.6363	9100	9.7762
0.6433	9200	9.1339
0.6503	9300	9.7895
0.6573	9400	9.657
0.6642	9500	8.4635
0.6712	9600	8.7074
0.6782	9700	8.6945
0.6852	9800	8.6175
0.6922	9900	8.2402
0.6992	10000	8.5626
0.7062	10100	8.3656
0.7132	10200	8.2415
0.7202	10300	8.0921
0.7272	10400	8.0562
0.7342	10500	8.2938
0.7412	10600	8.2436
0.7481	10700	8.5627
0.7551	10800	8.5165
0.7621	10900	8.6958
0.7691	11000	8.8117
0.7761	11100	9.1216
0.7831	11200	8.6971
0.7901	11300	9.3277
0.7971	11400	9.0345
0.8041	11500	8.4163
0.8111	11600	8.4963
0.8181	11700	8.2664
0.8251	11800	8.0255
0.8321	11900	8.4224
0.8390	12000	8.3533
0.8460	12100	8.0905
0.8530	12200	8.4241
0.8600	12300	8.3633
0.8670	12400	8.1362
0.8740	12500	8.4497
0.8810	12600	9.6622
0.8880	12700	8.4067
0.8950	12800	8.7043
0.9020	12900	8.5058
0.9090	13000	8.7286
0.9160	13100	8.3701
0.9229	13200	8.9587
0.9299	13300	8.3708
0.9369	13400	8.3968
0.9439	13500	8.6876
0.9509	13600	8.2832
0.9579	13700	8.4099
0.9649	13800	8.8301
0.9719	13900	8.6023
0.9789	14000	8.2473
0.9859	14100	8.8743
0.9929	14200	8.5779
0.9999	14300	8.3366
1.0069	14400	8.5381
1.0138	14500	8.2321
1.0208	14600	8.3483
1.0278	14700	8.0159
1.0348	14800	8.3423
1.0418	14900	8.5768
1.0488	15000	8.2451
1.0558	15100	8.5089
1.0628	15200	8.1973
1.0698	15300	7.9692
1.0768	15400	7.9994
1.0838	15500	7.8867
1.0908	15600	8.0215
1.0977	15700	8.0387
1.1047	15800	8.2535
1.1117	15900	8.221
1.1187	16000	8.228
1.1257	16100	8.2775
1.1327	16200	8.1873
1.1397	16300	8.2097
1.1467	16400	8.3559
1.1537	16500	8.1405
1.1607	16600	8.0454
1.1677	16700	8.4419
1.1747	16800	8.0014
1.1817	16900	8.2765
1.1886	17000	8.3162
1.1956	17100	8.5472
1.2026	17200	8.0504
1.2096	17300	7.9872
1.2166	17400	8.4175
1.2236	17500	8.4016
1.2306	17600	7.9588
1.2376	17700	8.3694
1.2446	17800	8.4478
1.2516	17900	8.365
1.2586	18000	8.6435
1.2656	18100	8.2626
1.2725	18200	8.1302
1.2795	18300	8.2361
1.2865	18400	8.2836
1.2935	18500	8.0584
1.3005	18600	8.0183
1.3075	18700	8.3594
1.3145	18800	8.3301
1.3215	18900	8.2401
1.3285	19000	8.1612
1.3355	19100	8.2225
1.3425	19200	8.2603
1.3495	19300	8.1074
1.3565	19400	8.3617
1.3634	19500	8.3055
1.3704	19600	8.329
1.3774	19700	8.2428
1.3844	19800	8.1095
1.3914	19900	8.2461
1.3984	20000	8.4066
1.4054	20100	8.1236
1.4124	20200	7.9104
1.4194	20300	7.9155
1.4264	20400	8.29
1.4334	20500	7.9682
1.4404	20600	8.1022
1.4474	20700	8.2172
1.4543	20800	7.8424
1.4613	20900	8.0254
1.4683	21000	7.8858
1.4753	21100	8.408
1.4823	21200	8.0559
1.4893	21300	8.0546
1.4963	21400	8.0694
1.5033	21500	8.1242
1.5103	21600	7.8775
1.5173	21700	8.394
1.5243	21800	8.5866
1.5313	21900	8.0003
1.5382	22000	7.9813
1.5452	22100	8.647
1.5522	22200	8.3267
1.5592	22300	7.9852
1.5662	22400	7.9187
1.5732	22500	8.2411
1.5802	22600	7.9273
1.5872	22700	8.0118
1.5942	22800	8.1991
1.6012	22900	7.9057
1.6082	23000	7.9906
1.6152	23100	8.4478
1.6222	23200	8.3375
1.6291	23300	8.5993
1.6361	23400	8.3598
1.6431	23500	8.0209
1.6501	23600	8.0557
1.6571	23700	7.8847
1.6641	23800	8.3735
1.6711	23900	8.4304
1.6781	24000	8.2099
1.6851	24100	8.2169
1.6921	24200	7.9044
1.6991	24300	8.1726
1.7061	24400	7.9407
1.7130	24500	8.3139
1.7200	24600	8.3866
1.7270	24700	7.9412
1.7340	24800	8.1111
1.7410	24900	8.4809
1.7480	25000	8.3234
1.7550	25100	8.0375
1.7620	25200	8.1444
1.7690	25300	7.9816
1.7760	25400	7.8436
1.7830	25500	8.2581
1.7900	25600	8.2902
1.7970	25700	8.0568
1.8039	25800	7.9311
1.8109	25900	8.0597
1.8179	26000	8.2886
1.8249	26100	8.2747
1.8319	26200	8.2165
1.8389	26300	8.3178
1.8459	26400	7.8966
1.8529	26500	8.0541
1.8599	26600	8.0102
1.8669	26700	8.2523
1.8739	26800	8.1137
1.8809	26900	8.2051
1.8878	27000	8.1109
1.8948	27100	7.9318
1.9018	27200	7.7605
1.9088	27300	8.2395
1.9158	27400	8.0824
1.9228	27500	8.1089
1.9298	27600	8.1654
1.9368	27700	8.0581
1.9438	27800	8.3029
1.9508	27900	8.0532
1.9578	28000	7.8507
1.9648	28100	7.9491
1.9718	28200	8.489
1.9787	28300	8.58
1.9857	28400	8.2852
1.9927	28500	7.9896
1.9997	28600	7.9989

Framework Versions

• Python: 3.8.10
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.4.1+cu118
• Accelerate: 1.0.1
• Datasets: 3.0.1
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

AnglELoss

@misc{li2023angleoptimized,
    title={AnglE-optimized Text Embeddings},
    author={Xianming Li and Jing Li},
    year={2023},
    eprint={2309.12871},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}