--- language: - en license: apache-2.0 tags: - sentence-transformers - sentence-similarity - feature-extraction - dense - generated_from_trainer - dataset_size:90000 - loss:CachedMultipleNegativesRankingLoss base_model: microsoft/mpnet-base widget: - source_sentence: how do i change the password for the app store? sentences: - Tap Settings > [your name] > Password & Security. Tap Change Password. Enter your current password or device passcode, then enter a new password and confirm the new password. Tap Change or Change Password. - Unless others attending the funeral know you for wearing bow ties, we recommend sticking to a necktie for funerals. We love bow ties, but they are statement pieces that can be a distraction at a funeral. ... Solid black bow ties, black tone-on-tone bow ties or dark solid color bow ties are the most appropriate choice. - 'Convert fraction (ratio) 5 / 35 Answer: 14.285714285714%' - source_sentence: how many molecules are present in one mole of a substance? sentences: - A year is divided into 12 months in the modern-day Gregorian calendar. The months are either 28, 29, 30, or 31 days long. Calendar with 12 months. - Latex is considered one of the best materials for mattress construction due to its natural softness, breathability and resiliency. Latex is most commonly found in luxury mattresses with high price points, and may be used in both the comfort layer and support core. - 'The mole allows scientists to calculate the number of elementary entities (usually atoms or molecules) in a certain mass of a given substance. Avogadro''s number is an absolute number: there are 6.022×1023 elementary entities in 1 mole. This can also be written as 6.022×1023 mol-1.' - source_sentence: how is baking powder and baking soda different? sentences: - While both products appear similar, they're certainly not the same. Baking soda is sodium bicarbonate, which requires an acid and a liquid to become activated and help baked goods rise. Conversely, baking powder includes sodium bicarbonate, as well as an acid. It only needs a liquid to become activated. - Donna Sheridan is dead, long live her hotel on the made-up Greek island of Kalokairi. You can rest assured that Streep, who still appears in new scenes in the movie, was fine with the decision — and according to director Ol Parker, it wasn't because she had other scheduling obligations. - Gyan (Sanskrit), a Sanskrit word that roughly translates to 'knowledge' in English. - source_sentence: what are the differences between food webs and food chains? sentences: - FOOD WEBS show how plants and animals are connected in many ways to help them all survive. FOOD CHAINS follow just one path of energy as animals find food. - 'The following instructions apply to making a PDF text-searchable in Adobe Acrobat Professional or Standard: Click on Tools > Text Recognition > In This File. The Recognize Text popup box opens. Select All pages, then click OK.' - Most sets are worth less than the $5 retail, though some like Civil War can command over $10, and the Aliens set of three are on eBay stores for $50. The rare US version of the action fleet Aliens jump ship has been asking over $70. - source_sentence: can you get period pains while pregnant? sentences: - The most confusing part about SACs is that your SAC score DOES NOT go towards your study score, at least not directly. In fact, your SAC score is merely used to rank you in your class. - For an uncomplicated UTI that occurs when you're otherwise healthy, your doctor may recommend a shorter course of treatment, such as taking an antibiotic for one to three days. But whether this short course of treatment is enough to treat your infection depends on your particular symptoms and medical history. - 'Some women will experience more cramping as they start to go into menopause. Pregnancy: Early in pregnancy, you may experience mild or light cramping. These cramps will probably feel like the light cramps you get during your period, but they''ll be in your lower stomach or lower back.' datasets: - sentence-transformers/gooaq pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: MPNet base with Landmark Pooling trained on GooAQ triplets using CachedMultipleNegativesRankingLoss with GradCache results: - task: type: information-retrieval name: Information Retrieval dataset: name: gooaq dev type: gooaq-dev metrics: - type: cosine_accuracy@1 value: 0.7581 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.896 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9329 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9641 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7581 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.29866666666666664 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.18658 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09641000000000001 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7581 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.896 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9329 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9641 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8654027716772075 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8332606746031707 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8349243908994848 name: Cosine Map@100 --- # MPNet base with Landmark Pooling trained on GooAQ triplets using CachedMultipleNegativesRankingLoss with GradCache This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [microsoft/mpnet-base](https://huggingface.co/microsoft/mpnet-base) on the [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [microsoft/mpnet-base](https://huggingface.co/microsoft/mpnet-base) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'splitter_type': 'variable', 'splitter_granularity': [32, 64, 128, 256], 'lmk_token_id': 2, 'architecture': 'MPNetModel'}) (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'pooling_mode_lmk': True, 'lmk_token_id': -1, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("tomaarsen/mpnet-base-gooaq-cmnrl-1024bs-lmk-gran-per-batch") # Run inference queries = [ "can you get period pains while pregnant?", ] documents = [ "Some women will experience more cramping as they start to go into menopause. Pregnancy: Early in pregnancy, you may experience mild or light cramping. These cramps will probably feel like the light cramps you get during your period, but they'll be in your lower stomach or lower back.", "For an uncomplicated UTI that occurs when you're otherwise healthy, your doctor may recommend a shorter course of treatment, such as taking an antibiotic for one to three days. But whether this short course of treatment is enough to treat your infection depends on your particular symptoms and medical history.", 'The most confusing part about SACs is that your SAC score DOES NOT go towards your study score, at least not directly. In fact, your SAC score is merely used to rank you in your class.', ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) print(query_embeddings.shape, document_embeddings.shape) # [1, 768] [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) # tensor([[ 0.7430, 0.1732, -0.0864]]) ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `gooaq-dev` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.7581 | | cosine_accuracy@3 | 0.896 | | cosine_accuracy@5 | 0.9329 | | cosine_accuracy@10 | 0.9641 | | cosine_precision@1 | 0.7581 | | cosine_precision@3 | 0.2987 | | cosine_precision@5 | 0.1866 | | cosine_precision@10 | 0.0964 | | cosine_recall@1 | 0.7581 | | cosine_recall@3 | 0.896 | | cosine_recall@5 | 0.9329 | | cosine_recall@10 | 0.9641 | | **cosine_ndcg@10** | **0.8654** | | cosine_mrr@10 | 0.8333 | | cosine_map@100 | 0.8349 | ## Training Details ### Training Dataset #### gooaq * Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c) * Size: 90,000 training samples * Columns: question and answer * Approximate statistics based on the first 1000 samples: | | question | answer | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | question | answer | |:--------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | how long does halifax take to transfer mortgage funds? | Bear in mind that the speed of application will vary depending on your own personal circumstances and the lender's present day-to-day performance. In some cases, applications can be approved by the lender within 24 hours, while some can take weeks or even months. | | can you get a false pregnancy test? | In very rare cases, you can have a false-positive result. This means you're not pregnant but the test says you are. You could have a false-positive result if you have blood or protein in your pee. Certain drugs, such as tranquilizers, anticonvulsants, hypnotics, and fertility drugs, could cause false-positive results. | | are ahead of its time? | Definition of ahead of one's/its time : too advanced or modern to be understood or appreciated during the time when one lives or works As a director, he was ahead of his time. | * Loss: [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim", "mini_batch_size": 64, "gather_across_devices": false } ``` ### Evaluation Dataset #### gooaq * Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c) * Size: 10,000 evaluation samples * Columns: question and answer * Approximate statistics based on the first 1000 samples: | | question | answer | |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | question | answer | |:-----------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | should you take ibuprofen with high blood pressure? | In general, people with high blood pressure should use acetaminophen or possibly aspirin for over-the-counter pain relief. Unless your health care provider has said it's OK, you should not use ibuprofen, ketoprofen, or naproxen sodium. If aspirin or acetaminophen doesn't help with your pain, call your doctor. | | how old do you have to be to work in sc? | The general minimum age of employment for South Carolina youth is 14, although the state allows younger children who are performers to work in show business. If their families are agricultural workers, children younger than age 14 may also participate in farm labor. | | how to write a topic proposal for a research paper? | ['Write down the main topic of your paper. ... ', 'Write two or three short sentences under the main topic that explain why you chose that topic. ... ', 'Write a thesis sentence that states the angle and purpose of your research paper. ... ', 'List the items you will cover in the body of the paper that support your thesis statement.'] | * Loss: [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim", "mini_batch_size": 64, "gather_across_devices": false } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 1024 - `per_device_eval_batch_size`: 1024 - `learning_rate`: 8e-05 - `num_train_epochs`: 1 - `warmup_ratio`: 0.1 - `warmup_steps`: 0.1 - `bf16`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 1024 - `per_device_eval_batch_size`: 1024 - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 8e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_ratio`: 0.1 - `warmup_steps`: 0.1 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `enable_jit_checkpoint`: False - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `use_cpu`: False - `seed`: 42 - `data_seed`: None - `bf16`: True - `fp16`: False - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: -1 - `ddp_backend`: None - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `group_by_length`: False - `length_column_name`: length - `project`: huggingface - `trackio_space_id`: trackio - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `hub_revision`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `auto_find_batch_size`: False - `full_determinism`: False - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_num_input_tokens_seen`: no - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `liger_kernel_config`: None - `eval_use_gather_object`: False - `average_tokens_across_devices`: True - `use_cache`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | Validation Loss | gooaq-dev_cosine_ndcg@10 | |:------:|:----:|:-------------:|:---------------:|:------------------------:| | -1 | -1 | - | - | 0.2180 | | 0.0114 | 1 | 6.3995 | - | - | | 0.0568 | 5 | 5.8563 | - | - | | 0.1023 | 9 | - | 1.1550 | 0.6291 | | 0.1136 | 10 | 2.4891 | - | - | | 0.1705 | 15 | 1.0676 | - | - | | 0.2045 | 18 | - | 0.5168 | 0.7867 | | 0.2273 | 20 | 0.7828 | - | - | | 0.2841 | 25 | 0.6288 | - | - | | 0.3068 | 27 | - | 0.3813 | 0.8246 | | 0.3409 | 30 | 0.5358 | - | - | | 0.3977 | 35 | 0.4802 | - | - | | 0.4091 | 36 | - | 0.3268 | 0.8399 | | 0.4545 | 40 | 0.4491 | - | - | | 0.5114 | 45 | 0.4258 | 0.2941 | 0.8490 | | 0.5682 | 50 | 0.4255 | - | - | | 0.6136 | 54 | - | 0.2762 | 0.8572 | | 0.625 | 55 | 0.4056 | - | - | | 0.6818 | 60 | 0.3826 | - | - | | 0.7159 | 63 | - | 0.2675 | 0.8580 | | 0.7386 | 65 | 0.3455 | - | - | | 0.7955 | 70 | 0.3722 | - | - | | 0.8182 | 72 | - | 0.2540 | 0.8631 | | 0.8523 | 75 | 0.3543 | - | - | | 0.9091 | 80 | 0.3519 | - | - | | 0.9205 | 81 | - | 0.2502 | 0.8633 | | 0.9659 | 85 | 0.3422 | - | - | | -1 | -1 | - | - | 0.8654 | ### Framework Versions - Python: 3.11.6 - Sentence Transformers: 5.3.0.dev0 - Transformers: 5.0.1.dev0 - PyTorch: 2.10.0+cu126 - Accelerate: 1.12.0 - Datasets: 4.3.0 - Tokenizers: 0.22.2 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### CachedMultipleNegativesRankingLoss ```bibtex @misc{gao2021scaling, title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan}, year={2021}, eprint={2101.06983}, archivePrefix={arXiv}, primaryClass={cs.LG} } ```