metadata
base_model: jet-taekyo/snowflake_finetuned_semantic
library_name: sentence-transformers
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
- dot_accuracy@1
- dot_accuracy@3
- dot_accuracy@5
- dot_accuracy@10
- dot_precision@1
- dot_precision@3
- dot_precision@5
- dot_precision@10
- dot_recall@1
- dot_recall@3
- dot_recall@5
- dot_recall@10
- dot_ndcg@10
- dot_mrr@10
- dot_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:714
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
- llama-cpp
- gguf-my-repo
widget:
- source_sentence: What are some examples of data privacy issues mentioned in the context?
sentences:
- >-
on a principle of local control, such that those individuals closest to
the data subject have more access while
those who are less proximate do not (e.g., a teacher has access to their
students’ daily progress data while a
superintendent does not).
Reporting. In addition to the reporting on data privacy (as listed above
for non-sensitive data), entities devel-
oping technologies related to a sensitive domain and those collecting,
using, storing, or sharing sensitive data
should, whenever appropriate, regularly provide public reports
describing: any data security lapses or breaches
that resulted in sensitive data leaks; the number, type, and outcomes of
ethical pre-reviews undertaken; a
description of any data sold, shared, or made public, and how that data
was assessed to determine it did not pres-
ent a sensitive data risk; and ongoing risk identification and
management procedures, and any mitigation added
- >-
DATA PRIVACY
HOW THESE PRINCIPLES CAN MOVE INTO PRACTICE
Real-life examples of how these principles can become reality, through
laws, policies, and practical
technical and sociotechnical approaches to protecting rights,
opportunities, and access.
The Privacy Act of 1974 requires privacy protections for personal
information in federal
records systems, including limits on data retention, and also provides
individuals a general
right to access and correct their data. Among other things, the Privacy
Act limits the storage of individual
information in federal systems of records, illustrating the principle of
limiting the scope of data retention. Under
the Privacy Act, federal agencies may only retain data about an
individual that is “relevant and necessary” to
accomplish an agency’s statutory purpose or to comply with an Executive
Order of the President. The law allows
- >-
DATA PRIVACY
WHY THIS PRINCIPLE IS IMPORTANT
This section provides a brief summary of the problems which the
principle seeks to address and protect
against, including illustrative examples.
•
An insurer might collect data from a person's social media presence as
part of deciding what life
insurance rates they should be offered.64
•
A data broker harvested large amounts of personal data and then suffered
a breach, exposing hundreds of
thousands of people to potential identity theft. 65
•
A local public housing authority installed a facial recognition system
at the entrance to housing complexes to
assist law enforcement with identifying individuals viewed via camera
when police reports are filed, leading
the community, both those living in the housing complex and not, to have
videos of them sent to the local
police department and made available for scanning by its facial
recognition software.66
•
- source_sentence: >-
What are the main topics covered in the National Institute of Standards
and Technology's AI Risk Management Framework?
sentences:
- >-
https://www.rand.org/pubs/research_reports/RRA2977-2.html.
Nicoletti, L. et al. (2023) Humans Are Biased. Generative Ai Is Even
Worse. Bloomberg.
https://www.bloomberg.com/graphics/2023-generative-ai-bias/.
National Institute of Standards and Technology (2024) Adversarial
Machine Learning: A Taxonomy and
Terminology of Attacks and Mitigations
https://csrc.nist.gov/pubs/ai/100/2/e2023/final
National Institute of Standards and Technology (2023) AI Risk Management
Framework.
https://www.nist.gov/itl/ai-risk-management-framework
National Institute of Standards and Technology (2023) AI Risk Management
Framework, Chapter 3: AI
Risks and Trustworthiness.
https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics
National Institute of Standards and Technology (2023) AI Risk Management
Framework, Chapter 6: AI
RMF Profiles.
https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile
- >-
(e.g., via red-teaming, field testing, participatory engagements,
performance
assessments, user feedback mechanisms).
Human-AI Configuration
AI Actor Tasks: AI Development, AI Deployment, AI Impact Assessment,
Operation and Monitoring
MANAGE 2.2: Mechanisms are in place and applied to sustain the value of
deployed AI systems.
Action ID
Suggested Action
GAI Risks
MG-2.2-001
Compare GAI system outputs against pre-defined organization risk
tolerance,
guidelines, and principles, and review and test AI-generated content
against
these guidelines.
CBRN Information or Capabilities;
Obscene, Degrading, and/or
Abusive Content; Harmful Bias and
Homogenization; Dangerous,
Violent, or Hateful Content
MG-2.2-002
Document training data sources to trace the origin and provenance of AI-
generated content.
Information Integrity
MG-2.2-003
Evaluate feedback loops between GAI system content provenance and human
- >-
domain or for functions that are required for administrative reasons
(e.g., school attendance records), unless
consent is acquired, if appropriate, and the additional expectations in
this section are met. Consent for non-
necessary functions should be optional, i.e., should not be required,
incentivized, or coerced in order to
receive opportunities or access to services. In cases where data is
provided to an entity (e.g., health insurance
company) in order to facilitate payment for such a need, that data
should only be used for that purpose.
Ethical review and use prohibitions. Any use of sensitive data or
decision process based in part on sensi-
tive data that might limit rights, opportunities, or access, whether the
decision is automated or not, should go
through a thorough ethical review and monitoring, both in advance and by
periodic review (e.g., via an indepen-
dent ethics committee or similarly robust process). In some cases, this
ethical review may determine that data
- source_sentence: >-
How can organizations leverage user feedback to enhance content provenance
and risk management efforts?
sentences:
- >-
tested, there will always be situations for which the system fails. The
American public deserves protection via human
review against these outlying or unexpected scenarios. In the case of
time-critical systems, the public should not have
to wait—immediate human consideration and fallback should be available.
In many time-critical systems, such a
remedy is already immediately available, such as a building manager who
can open a door in the case an automated
card access system fails.
In the criminal justice system, employment, education, healthcare, and
other sensitive domains, automated systems
are used for many purposes, from pre-trial risk assessments and parole
decisions to technologies that help doctors
diagnose disease. Absent appropriate safeguards, these technologies can
lead to unfair, inaccurate, or dangerous
outcomes. These sensitive domains require extra protections. It is
critically important that there is extensive human
oversight in such settings.
- >-
enable organizations to maximize the utility of provenance data and risk
management efforts.
A.1.7. Enhancing Content Provenance through Structured Public Feedback
While indirect feedback methods such as automated error collection
systems are useful, they often lack
the context and depth that direct input from end users can provide.
Organizations can leverage feedback
approaches described in the Pre-Deployment Testing section to capture
input from external sources such
as through AI red-teaming.
Integrating pre- and post-deployment external feedback into the
monitoring process for GAI models and
corresponding applications can help enhance awareness of performance
changes and mitigate potential
risks and harms from outputs. There are many ways to capture and make
use of user feedback – before
and after GAI systems and digital content transparency approaches are
deployed – to gain insights about
- >-
A.1. Governance
A.1.1. Overview
Like any other technology system, governance principles and techniques
can be used to manage risks
related to generative AI models, capabilities, and applications.
Organizations may choose to apply their
existing risk tiering to GAI systems, or they may opt to revise or
update AI system risk levels to address
these unique GAI risks. This section describes how organizational
governance regimes may be re-
evaluated and adjusted for GAI contexts. It also addresses third-party
considerations for governing across
the AI value chain.
A.1.2. Organizational Governance
GAI opportunities, risks and long-term performance characteristics are
typically less well-understood
than non-generative AI tools and may be perceived and acted upon by
humans in ways that vary greatly.
Accordingly, GAI may call for different levels of oversight from AI
Actors or different human-AI
- source_sentence: >-
What should be ensured for users who have trouble with the automated
system?
sentences:
- >-
32
MEASURE 2.6: The AI system is evaluated regularly for safety risks – as
identified in the MAP function. The AI system to be
deployed is demonstrated to be safe, its residual negative risk does not
exceed the risk tolerance, and it can fail safely, particularly if
made to operate beyond its knowledge limits. Safety metrics reflect
system reliability and robustness, real-time monitoring, and
response times for AI system failures.
Action ID
Suggested Action
GAI Risks
MS-2.6-001
Assess adverse impacts, including health and wellbeing impacts for value
chain
or other AI Actors that are exposed to sexually explicit, offensive, or
violent
information during GAI training and maintenance.
Human-AI Configuration; Obscene,
Degrading, and/or Abusive
Content; Value Chain and
Component Integration;
Dangerous, Violent, or Hateful
Content
MS-2.6-002
Assess existence or levels of harmful bias, intellectual property
infringement,
- >-
APPENDIX
Systems that impact the safety of communities such as automated traffic
control systems, elec
-ctrical grid controls, smart city technologies, and industrial
emissions and environmental
impact control algorithms; and
Systems related to access to benefits or services or assignment of
penalties such as systems that
support decision-makers who adjudicate benefits such as collating or
analyzing information or
matching records, systems which similarly assist in the adjudication of
administrative or criminal
penalties, fraud detection algorithms, services or benefits access
control algorithms, biometric
systems used as access control, and systems which make benefits or
services related decisions on a
fully or partially autonomous basis (such as a determination to revoke
benefits).
54
- >-
meaningfully impact rights, opportunities, or access should have greater
availability (e.g., staffing) and over
sight of human consideration and fallback mechanisms.
Accessible. Mechanisms for human consideration and fallback, whether
in-person, on paper, by phone, or
otherwise provided, should be easy to find and use. These mechanisms
should be tested to ensure that users
who have trouble with the automated system are able to use human
consideration and fallback, with the under
standing that it may be these users who are most likely to need the
human assistance. Similarly, it should be
tested to ensure that users with disabilities are able to find and use
human consideration and fallback and also
request reasonable accommodations or modifications.
Convenient. Mechanisms for human consideration and fallback should not
be unreasonably burdensome as
compared to the automated system’s equivalent.
49
- source_sentence: >-
What must lenders provide to consumers who are denied credit under the
Fair Credit Reporting Act?
sentences:
- >-
8
Trustworthy AI Characteristics: Accountable and Transparent, Privacy
Enhanced, Safe, Secure and
Resilient
2.5. Environmental Impacts
Training, maintaining, and operating (running inference on) GAI systems
are resource-intensive activities,
with potentially large energy and environmental footprints. Energy and
carbon emissions vary based on
what is being done with the GAI model (i.e., pre-training, fine-tuning,
inference), the modality of the
content, hardware used, and type of task or application.
Current estimates suggest that training a single transformer LLM can
emit as much carbon as 300 round-
trip flights between San Francisco and New York. In a study comparing
energy consumption and carbon
emissions for LLM inference, generative tasks (e.g., text summarization)
were found to be more energy-
and carbon-intensive than discriminative or non-generative tasks (e.g.,
text classification).
- >-
that consumers who are denied credit receive "adverse action" notices.
Anyone who relies on the information in a
credit report to deny a consumer credit must, under the Fair Credit
Reporting Act, provide an "adverse action"
notice to the consumer, which includes "notice of the reasons a creditor
took adverse action on the application
or on an existing credit account."90 In addition, under the risk-based
pricing rule,91 lenders must either inform
borrowers of their credit score, or else tell consumers when "they are
getting worse terms because of
information in their credit report." The CFPB has also asserted that
"[t]he law gives every applicant the right to
a specific explanation if their application for credit was denied, and
that right is not diminished simply because
a company uses a complex algorithm that it doesn't understand."92 Such
explanations illustrate a shared value
that certain decisions need to be explained.
- >-
measures to prevent, flag, or take other action in response to outputs
that
reproduce particular training data (e.g., plagiarized, trademarked,
patented,
licensed content or trade secret material).
Intellectual Property; CBRN
Information or Capabilities
model-index:
- name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-m
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: Unknown
type: unknown
metrics:
- type: cosine_accuracy@1
value: 0.875
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.9671052631578947
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.9868421052631579
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.993421052631579
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.875
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.3223684210526316
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.19736842105263155
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.09934210526315788
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.875
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.9671052631578947
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.9868421052631579
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.993421052631579
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.9420758802321664
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.9248903508771928
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.925488437001595
name: Cosine Map@100
- type: dot_accuracy@1
value: 0.875
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.9671052631578947
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.9868421052631579
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.993421052631579
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.875
name: Dot Precision@1
- type: dot_precision@3
value: 0.3223684210526316
name: Dot Precision@3
- type: dot_precision@5
value: 0.19736842105263155
name: Dot Precision@5
- type: dot_precision@10
value: 0.09934210526315788
name: Dot Precision@10
- type: dot_recall@1
value: 0.875
name: Dot Recall@1
- type: dot_recall@3
value: 0.9671052631578947
name: Dot Recall@3
- type: dot_recall@5
value: 0.9868421052631579
name: Dot Recall@5
- type: dot_recall@10
value: 0.993421052631579
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.9420758802321664
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.9248903508771928
name: Dot Mrr@10
- type: dot_map@100
value: 0.925488437001595
name: Dot Map@100
- type: cosine_accuracy@1
value: 0.890625
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.96875
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.96875
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.9765625
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.890625
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.32291666666666663
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.19375000000000003
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.09765625000000003
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.890625
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.96875
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.96875
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.9765625
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.9391060398540476
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.9265625
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.9282275883838385
name: Cosine Map@100
- type: dot_accuracy@1
value: 0.890625
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.96875
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.96875
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.9765625
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.890625
name: Dot Precision@1
- type: dot_precision@3
value: 0.32291666666666663
name: Dot Precision@3
- type: dot_precision@5
value: 0.19375000000000003
name: Dot Precision@5
- type: dot_precision@10
value: 0.09765625000000003
name: Dot Precision@10
- type: dot_recall@1
value: 0.890625
name: Dot Recall@1
- type: dot_recall@3
value: 0.96875
name: Dot Recall@3
- type: dot_recall@5
value: 0.96875
name: Dot Recall@5
- type: dot_recall@10
value: 0.9765625
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.9391060398540476
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.9265625
name: Dot Mrr@10
- type: dot_map@100
value: 0.9282275883838385
name: Dot Map@100
Sleem247/snowflake_finetuned_semantic-Q8_0-GGUF
This model was converted to GGUF format from jet-taekyo/snowflake_finetuned_semantic using llama.cpp via the ggml.ai's GGUF-my-repo space.
Refer to the original model card for more details on the model.
Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
brew install llama.cpp
Invoke the llama.cpp server or the CLI.
CLI:
llama-cli --hf-repo Sleem247/snowflake_finetuned_semantic-Q8_0-GGUF --hf-file snowflake_finetuned_semantic-q8_0.gguf -p "The meaning to life and the universe is"
Server:
llama-server --hf-repo Sleem247/snowflake_finetuned_semantic-Q8_0-GGUF --hf-file snowflake_finetuned_semantic-q8_0.gguf -c 2048
Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.
Step 1: Clone llama.cpp from GitHub.
git clone https://github.com/ggerganov/llama.cpp
Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
cd llama.cpp && LLAMA_CURL=1 make
Step 3: Run inference through the main binary.
./llama-cli --hf-repo Sleem247/snowflake_finetuned_semantic-Q8_0-GGUF --hf-file snowflake_finetuned_semantic-q8_0.gguf -p "The meaning to life and the universe is"
or
./llama-server --hf-repo Sleem247/snowflake_finetuned_semantic-Q8_0-GGUF --hf-file snowflake_finetuned_semantic-q8_0.gguf -c 2048