Add new SentenceTransformer model

936b930 verified about 1 month ago

26.3 kB

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:90000
  - loss:CachedMultipleNegativesRankingLoss
base_model: microsoft/mpnet-base
widget:
  - source_sentence: how do i change the password for the app store?
    sentences:
      - >-
        Tap Settings > [your name] > Password & Security. Tap Change Password.
        Enter your current password or device passcode, then enter a new
        password and confirm the new password. Tap Change or Change Password.
      - >-
        Unless others attending the funeral know you for wearing bow ties, we
        recommend sticking to a necktie for funerals. We love bow ties, but they
        are statement pieces that can be a distraction at a funeral. ... Solid
        black bow ties, black tone-on-tone bow ties or dark solid color bow ties
        are the most appropriate choice.
      - 'Convert fraction (ratio) 5 / 35 Answer: 14.285714285714%'
  - source_sentence: how many molecules are present in one mole of a substance?
    sentences:
      - >-
        A year is divided into 12 months in the modern-day Gregorian calendar.
        The months are either 28, 29, 30, or 31 days long. Calendar with 12
        months.
      - >-
        Latex is considered one of the best materials for mattress construction
        due to its natural softness, breathability and resiliency. Latex is most
        commonly found in luxury mattresses with high price points, and may be
        used in both the comfort layer and support core.
      - >-
        The mole allows scientists to calculate the number of elementary
        entities (usually atoms or molecules) in a certain mass of a given
        substance. Avogadro's number is an absolute number: there are 6.022×1023
        elementary entities in 1 mole. This can also be written as 6.022×1023
        mol-1.
  - source_sentence: how is baking powder and baking soda different?
    sentences:
      - >-
        While both products appear similar, they're certainly not the same.
        Baking soda is sodium bicarbonate, which requires an acid and a liquid
        to become activated and help baked goods rise. Conversely, baking powder
        includes sodium bicarbonate, as well as an acid. It only needs a liquid
        to become activated.
      - >-
        Donna Sheridan is dead, long live her hotel on the made-up Greek island
        of Kalokairi. You can rest assured that Streep, who still appears in new
        scenes in the movie, was fine with the decision — and according to
        director Ol Parker, it wasn't because she had other scheduling
        obligations.
      - >-
        Gyan (Sanskrit), a Sanskrit word that roughly translates to 'knowledge'
        in English.
  - source_sentence: what are the differences between food webs and food chains?
    sentences:
      - >-
        FOOD WEBS show how plants and animals are connected in many ways to help
        them all survive. FOOD CHAINS follow just one path of energy as animals
        find food.
      - >-
        The following instructions apply to making a PDF text-searchable in
        Adobe Acrobat Professional or Standard: Click on Tools > Text
        Recognition > In This File. The Recognize Text popup box opens. Select
        All pages, then click OK.
      - >-
        Most sets are worth less than the $5 retail, though some like Civil War
        can command over $10, and the Aliens set of three are on eBay stores for
        $50. The rare US version of the action fleet Aliens jump ship has been
        asking over $70.
  - source_sentence: can you get period pains while pregnant?
    sentences:
      - >-
        The most confusing part about SACs is that your SAC score DOES NOT go
        towards your study score, at least not directly. In fact, your SAC score
        is merely used to rank you in your class.
      - >-
        For an uncomplicated UTI that occurs when you're otherwise healthy, your
        doctor may recommend a shorter course of treatment, such as taking an
        antibiotic for one to three days. But whether this short course of
        treatment is enough to treat your infection depends on your particular
        symptoms and medical history.
      - >-
        Some women will experience more cramping as they start to go into
        menopause. Pregnancy: Early in pregnancy, you may experience mild or
        light cramping. These cramps will probably feel like the light cramps
        you get during your period, but they'll be in your lower stomach or
        lower back.
datasets:
  - sentence-transformers/gooaq
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: >-
      MPNet base with Landmark Pooling trained on GooAQ triplets using
      CachedMultipleNegativesRankingLoss with GradCache
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: gooaq dev
          type: gooaq-dev
        metrics:
          - type: cosine_accuracy@1
            value: 0.7581
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.896
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9329
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9641
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7581
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.29866666666666664
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.18658
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09641000000000001
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7581
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.896
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9329
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9641
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8654027716772075
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8332606746031707
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.8349243908994848
            name: Cosine Map@100

MPNet base with Landmark Pooling trained on GooAQ triplets using CachedMultipleNegativesRankingLoss with GradCache

This is a sentence-transformers model finetuned from microsoft/mpnet-base on the gooaq dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: microsoft/mpnet-base
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- gooaq
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'splitter_type': 'variable', 'splitter_granularity': [32, 64, 128, 256], 'lmk_token_id': 2, 'architecture': 'MPNetModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'pooling_mode_lmk': True, 'lmk_token_id': -1, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/mpnet-base-gooaq-cmnrl-1024bs-lmk-gran-per-batch")
# Run inference
queries = [
    "can you get period pains while pregnant?",
]
documents = [
    "Some women will experience more cramping as they start to go into menopause. Pregnancy: Early in pregnancy, you may experience mild or light cramping. These cramps will probably feel like the light cramps you get during your period, but they'll be in your lower stomach or lower back.",
    "For an uncomplicated UTI that occurs when you're otherwise healthy, your doctor may recommend a shorter course of treatment, such as taking an antibiotic for one to three days. But whether this short course of treatment is enough to treat your infection depends on your particular symptoms and medical history.",
    'The most confusing part about SACs is that your SAC score DOES NOT go towards your study score, at least not directly. In fact, your SAC score is merely used to rank you in your class.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.7430,  0.1732, -0.0864]])

Evaluation

Metrics

Information Retrieval

Dataset: gooaq-dev
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7581
cosine_accuracy@3	0.896
cosine_accuracy@5	0.9329
cosine_accuracy@10	0.9641
cosine_precision@1	0.7581
cosine_precision@3	0.2987
cosine_precision@5	0.1866
cosine_precision@10	0.0964
cosine_recall@1	0.7581
cosine_recall@3	0.896
cosine_recall@5	0.9329
cosine_recall@10	0.9641
cosine_ndcg@10	0.8654
cosine_mrr@10	0.8333
cosine_map@100	0.8349

Training Details

Training Dataset

gooaq

Dataset: gooaq at b089f72
Size: 90,000 training samples
Columns: question and answer
Approximate statistics based on the first 1000 samples:
question answer
type string string
details
min: 8 tokens
mean: 11.83 tokens
max: 20 tokens

min: 15 tokens
mean: 61.75 tokens
max: 185 tokens

	question	answer
type	string	string
details	min: 8 tokens mean: 11.83 tokens max: 20 tokens	min: 15 tokens mean: 61.75 tokens max: 185 tokens

Samples:

question	answer
`how long does halifax take to transfer mortgage funds?`	`Bear in mind that the speed of application will vary depending on your own personal circumstances and the lender's present day-to-day performance. In some cases, applications can be approved by the lender within 24 hours, while some can take weeks or even months.`
`can you get a false pregnancy test?`	`In very rare cases, you can have a false-positive result. This means you're not pregnant but the test says you are. You could have a false-positive result if you have blood or protein in your pee. Certain drugs, such as tranquilizers, anticonvulsants, hypnotics, and fertility drugs, could cause false-positive results.`
`are ahead of its time?`	`Definition of ahead of one's/its time : too advanced or modern to be understood or appreciated during the time when one lives or works As a director, he was ahead of his time.`

Loss: CachedMultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "mini_batch_size": 64,
    "gather_across_devices": false
}

Evaluation Dataset

gooaq

Dataset: gooaq at b089f72
Size: 10,000 evaluation samples
Columns: question and answer
Approximate statistics based on the first 1000 samples:
question answer
type string string
details
min: 8 tokens
mean: 11.93 tokens
max: 25 tokens

min: 14 tokens
mean: 61.2 tokens
max: 128 tokens

	question	answer
type	string	string
details	min: 8 tokens mean: 11.93 tokens max: 25 tokens	min: 14 tokens mean: 61.2 tokens max: 128 tokens

Samples:

question	answer
`should you take ibuprofen with high blood pressure?`	`In general, people with high blood pressure should use acetaminophen or possibly aspirin for over-the-counter pain relief. Unless your health care provider has said it's OK, you should not use ibuprofen, ketoprofen, or naproxen sodium. If aspirin or acetaminophen doesn't help with your pain, call your doctor.`
`how old do you have to be to work in sc?`	`The general minimum age of employment for South Carolina youth is 14, although the state allows younger children who are performers to work in show business. If their families are agricultural workers, children younger than age 14 may also participate in farm labor.`
`how to write a topic proposal for a research paper?`	`['Write down the main topic of your paper. ... ', 'Write two or three short sentences under the main topic that explain why you chose that topic. ... ', 'Write a thesis sentence that states the angle and purpose of your research paper. ... ', 'List the items you will cover in the body of the paper that support your thesis statement.']`

Loss: CachedMultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "mini_batch_size": 64,
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 1024
per_device_eval_batch_size: 1024
learning_rate: 8e-05
num_train_epochs: 1
warmup_ratio: 0.1
warmup_steps: 0.1
bf16: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 1024
per_device_eval_batch_size: 1024
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 8e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: 0.1
warmup_steps: 0.1
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooaq-dev_cosine_ndcg@10
-1	-1	-	-	0.2180
0.0114	1	6.3995	-	-
0.0568	5	5.8563	-	-
0.1023	9	-	1.1550	0.6291
0.1136	10	2.4891	-	-
0.1705	15	1.0676	-	-
0.2045	18	-	0.5168	0.7867
0.2273	20	0.7828	-	-
0.2841	25	0.6288	-	-
0.3068	27	-	0.3813	0.8246
0.3409	30	0.5358	-	-
0.3977	35	0.4802	-	-
0.4091	36	-	0.3268	0.8399
0.4545	40	0.4491	-	-
0.5114	45	0.4258	0.2941	0.8490
0.5682	50	0.4255	-	-
0.6136	54	-	0.2762	0.8572
0.625	55	0.4056	-	-
0.6818	60	0.3826	-	-
0.7159	63	-	0.2675	0.8580
0.7386	65	0.3455	-	-
0.7955	70	0.3722	-	-
0.8182	72	-	0.2540	0.8631
0.8523	75	0.3543	-	-
0.9091	80	0.3519	-	-
0.9205	81	-	0.2502	0.8633
0.9659	85	0.3422	-	-
-1	-1	-	-	0.8654

Framework Versions

Python: 3.11.6
Sentence Transformers: 5.3.0.dev0
Transformers: 5.0.1.dev0
PyTorch: 2.10.0+cu126
Accelerate: 1.12.0
Datasets: 4.3.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}