AI Practitioner Exam - AI Aspects

   

AI Practitioner Exam Prep

AI Aspects

Cost Considerations

  Cost factors are responsiveness and availability, redundancy and regional coverage, performance, token-based pricing, provisioned throughput, and refining your custom models.

Responsible AI

  Aspects: fairness, explainability, privacy\security (theft and exposure risk), veracity\robustness (operates well despite uncertainty), governance (max society benefit with min risk), interpretability\transparency, safety, and controllability.

  Business Benefits: trust, regulatory complain, mitigate risks, competitive advantage, improved decision making, and improved products. 

  Model Selection: a) Narrow the use case so you can tune your model to your use case. Ex: favor recall or precision, b) pick by performance with some test data sets, c) responsible agency, d) environmental reasons, e) economical reasons.

  Dataset Prep: want balanced dataset so inclusive and diverse in data collection, curating by 1) preprocessing, 2) augmentation, and 3) regular auditing.

  Model Tradeoffs: a) interpretability vs. performance (wrong - see Lonestar), b) safety vs. transparency (wrong - see open source), c) control over the model.

  Human Centered Design: amplified decision making, unbiased decision making, RLHF.

Benchmarking Datasets

  Way 1: Humans evaluation is done by 1) human SMEs create questions. 2) context identified, 3) answer created. 

   Way 2: LLMs as judges approach grades looking at answers compared to benchmarking datasets. 

Security and Compliance in AI

  AWS supports 143 security standards and compliance certifications, such as GDPR, HIPPA, ISO, PCI DSS, etc.

   Security scopes: Consumer app, Enterprise app, Pre-trained models, Fine-tuned models, and self-trained models.

OWASP Top 10 AI Security risks

  Prompt injection: Malicious user inputs that can manipulate the behavior of a language model

  Insecure output handling: Fail to properly sanitize or validate model outputs.

  Training data poisoning: Introducing bad data into a model's training set, so bad behaviors.

  Model denial of service: Exploits vulnerabilities in a model's architecture to disrupt its availability

  Supply chain vulnerabilities: Weaknesses in the software, hardware, or services to build a model.

  Sensitive info disclosure: Leak sensitive data through model outputs or other unintended channels

  Insecure plugin design: Flaws in the optional model components that can be exploited

  Excessive agency: Grants a model too much autonomy or capability.

  Overreliance: Over-dependence on a model's capabilities.

  Model theft: Unauthorized access or copying of a model's parameters or architecture.

 

  Prompt template theft is possibility.

  Overall the risks are fake content, prompt injection, and AI model weaknesses.

  Overall to secure data, you should control user access to the data and ensure data integrity.

Data Governance 

  Strategies: data quality and integrity, data protection and security, data lifecycle mgmt., responsible AI, governance structures and roles, and data sharing and collection.

  Approaches: policies, review cadence, review strategies, transparency, and team training standards.

Misc

   In multi-step tasks, AI agents are important in task coordination such as task sequence. 

   Storing vector databases (such as custom ML models) is good for OpenSearch Service since is a fully managed service that supports vector data types, for storing and querying embeddings efficiently.

   Model pruning is reducing model size and complexity.