ML Exam: 2

Background

SageMaker AI

SageMaker AI is the “heart” of the MLA-C01 certification

The majority of exam questions will be with SageMaker.
It is important to understand and discern between SageMaker Processing, SageMaker Training, and SageMaker Hosting, which all cover different aspects of the end-to-end ML process.
These notes first cover generic ML knowledge and concepts, and then their implementation in AWS (usually involving SageMaker and other AWS services).
Some open-source Apache services like Hadoop or Spark are also covered, since they are also popular in ML environments and are well supported in AWS
It is a good idea to review the high-level overview of SageMaker that was done in the foundational AIF-C01 certification. MLA-C01 builds on top of that knowledge.

Intro to SageMaker AI

AWS service that can handle the whole End-to-End process in ML
- Data processing, model training, model deployment, and model hosting
- Tons of features and sub-products (will go into depth in these notes)
SageMaker Training and Deployment Architecture

Input/output data usually in S3, but could be in other data stores
Training and inference code must be inside container images registered in ECR
Not all ML models will be deployed to endpoints.

End-to-End Process:

Data Preparation (data prep)
- Data usually comes from S3
  - Data can also come from Athena, EMR, Redshift, Amazon Keyspaces DB…
- Integration with Apache Spark
Data Processing
- Processing job: copy raw data from S3 → Spin up processing container → Output processed data to S3
- Container can be SageMaker built-in or user provided (code)
Training
- Training job requires
  - URL of S3 bucket with training data
  - ML compute resources
  - URL of S3 bucket for output → Model outputted to S3
  - Container (ECR) path to training code
- Many training options available
  - Built-in algorithms, Spark MLLib, Tensorflow, PyTorch, Scikit-learn, XGBoost, Hugging Face, your own Docker image, AWS marketplace-purchased algorithms…
Deployment

2 ways:
- Persistent endpoint for individual predictions/inference on demand
- SageMaker Batch Transform for predictions of an entire dataset
Many cool options: inference pipelines, SageMaker Neo (edge devices), Elastic Inference, automatic scaling, shadow testing…

SageMaker Domain

Organizational unit within SageMaker → organize users, apps, and resources
- A domain must be configured before you can do anything in SageMaker!
  - Think of it as an isolation of an ML project
Each domain has one EFS volume
- Each user has their private EFS directory within that volume
- There's a shared EFS directory available to all users
User profile: represents an individual user/person in a domain
- Can create own personal apps
- Can spin private SageMaker Studio instances
- Has access to a private EFS directory to store personal files
Shared resources across all users:
- Shared spaces
- Shared EFS directory
- Communal IDE app (SageMaker Studio public to all users)

Network Configuration in SageMaker Domain

By default, a domain has two VPCs

One with public internet access → can expose public endpoints for your domain
- Managed by SageMaker
- Optional → can select “VPC Only” when creating the domain, which means this managed VPC is NOT created
One for private traffic

Encrypted traffic to domain's EFS volume
YOU manage it: must specify the VPC, its subnets, and security groups (SGs)

Interfaces for Using SageMaker

SageMaker Notebooks

Old/classic method for ML in SageMaker → ML code
Spin up EC2 instances to host ML Notebooks, which direct ML E2E process:
- S3 data access
- ML code in Jupyter Notebook
  - Libraries like Scikit_learn, numpy, pandas, Apache Spark, Tensorflow, etc at your disposal
  - Wide variety of built-in models
- Can spin up training instances
- Can deploy trained models for making predictions (inferring) at scale

SageMaker SDKs

Training and deployment of ML models via Python scripts
Python API libraries → import inside your code
1. Boto3 (low-level API)
2. SageMaker Python SDK (high-level API)
Can automate ML workflows, manage training jobs, deployments, and pipelines

SageMaker Studio

Web-based IDE for E2E ML development
Features: Team collaboration, Tune and debug ML models, Deploy ML models, Automated workflows.
Screenshot:

SageMaker Console UI

AWS Management Console interface for SageMaker
- GUI for managing SageMaker resources
Mostly for administrative tasks
Can access all other interfaces from the console UI
Screenshot:

SageMaker Jumpstart

ML Hub with many pre-trained ML models and pre-built ML solutions. Offers one-click deployment of models for inference. End-to-end solutions for common business problems.
Computer Vision (CV) models, Natural Language Processing (NLP) models, GenAI Foundation Models (FMs)…
- Amazon-owned models or 3rd-party provider models
- Provider examples: HuggingFace, Databricks, Meta…
Screenshot:

SageMaker Canvas (was Data Wrangler)

Canvas became the unified no-code workspace for both data prep and model building. Data Wrangler was integrated directly into Canvas. No-code ML for business analysts.
Features
- Build custom ML model (leverages AutoML powered by SageMaker Autopilot)
  - e.g. Upload CSV data (CSV-only for now), select column to predict & build model
- Automatic data cleaning (leverages Data Wrangler)
- Access ready-to-use models from AWS AI services (Rekognition, Comprehend…)
- GenAI support via Bedrock or JumpStart FMs
- Import, preview, visualize, transform data… in a visual UI
- - Even “Quick Model”
  - Can also export data flow
- Many feature engineering capabilities (transform images, balance data, impute missing data, handle outliers, PCA…)
- Troubleshooting:
Screenshot:

Summary Table of Interfaces for SageMaker

SageMaker Ground Truth

Humans label data → Prepare a training dataset with humans

Human reviewers:

Mechanical Turk workers, your employees, or third-party vendors. Ground Truth creates own model as humans label data → RLHF. Only images the model isn't sure about are sent to human labelers (reduces manual work by 70%).

Ground Truth Plus: Turnkey solution.
AWS experts manage the whole workflow. Fill out a form. Experts contact you, discuss pricing, manage labelers. Do NOT confuse with Amazon Augmented AI (A2I)!

Labels
GroundTruth is for human labeling, while A2I is for human oversight of trained model predictions. However, SageMaker Ground Truth and A2I can use the same human workforce! Benefits: consistency, efficiency, flexibility. Other ways: Rekognition, Comprehend, etc. Some pre-trained models or unsupervised techniques can be helpful.

SageMaker Feature Store

Centralized portal for features. Offers fast, secure access to feature data for ML models.
Data ingestion via streaming or batch
- Feature Store inputs data from streams with PutRecord API.
- Online store (Model gets features with GetRecord API)
- Offline store (Feature Store inputs data into S3, AWS Glue creates Data Catalog, models can then access features via BATCH access)
Security
- encryption at rest (KMS…) and in transit
- IAM, PrivateLink…

SageMaker Model Monitor

Detects data drift, model drift, and bias drift in deployed ML models. Continuously compares incoming data against a baseline dataset captured during training.
When detects drift beyond thresholds, it emits Amazon CloudWatch events. These events can trigger an AWS Lambda function, which is a common way to automate workflows such as model retraining. Then the lambda function starts a SageMaker Pipeline, firing a retraining job with updated data.

Feature Engineering

Feature Engineering Tools

* Sagemaker Data Wrangler
* Sagemaker Canvas.
These services offer over 300 built-in transformations.

Feature Engineering - Basic Concepts

Applying domain knowledge (your data knowledge and model knowledge) to create better features to train your model is the ART OF ML!!. Most critical part in a good ML implementation. Talented/expert ML specialists are good at feature engineering.
Curse of dimensionality is more features is not better! Every feature is a new dimension. Much of feature engineering is selecting most relevant features → domain knowledge comes into play. Unsupervised dimensionality reduction can help (PCA, K-Means).

Data

Types of Structure of Data

Properties of Data (4 Vs)

1. Volume (Size) - GBs? PBs?…

2. Velocity - High velocity → Real-Time or near-RT processing

3. Variety - Structured? Mixed? Multiple sources? Multiple formats?

4. Veracity?

Data Warehouses, Data Lakes, Data Lakehouses

- Data Warehouse (DWH) (e.g. Amazon Redshift) - Centralized repository optimized for analysis (read-heavy operations) where data from different sources is stored in a structured format

- Data Lake (e.g. Amazon S3 can be used as data lake) - Storage repository that holds vast amounts of raw data in its native format (predefined structure is not necessary). Structured, semi-structured, & unstructured data

- Often, organizations use a combination of both, ingesting raw data into a data lake and then processing and moving refined data into a data warehouse for analysis

- Data Lakehouse (e.g. AWS Lake Formation with S3 & Redshift Spectrum) - Hybrid data architecture, tries to provide advantages of both. Performance, reliability & capabilities of DWHs. Flexibility, scale & low-cost storage of data lakes.

Data Mesh

Domain-based data management paradigm. Decentralized architectural framework that shifts data ownership from a central team to domain-specific teams (e.g., marketing, sales, shipping).

Data Columns - Leakage

Leakage causes a model to look accurate during training and testing, but fails in production. Because the ML will just decide to only look at that column for the answer, since the column is perfectly matches the prediction.

Examples:

1) "Airbag": If predicting safe drivers, and training data has Airbag Deployed column, Canvas/Data Wrangler will flag Airbag Deployed as target leakage. In production, you need to predict the crash before it happens, at which point the airbag has not yet deployed.
2) "Customer Churn": If predicting Customer Churn and have Account Cancellation Date column, this will have "Yes" values. It is a leaked feature, since no cancellation date for active customer you are trying to retain.

3) "Transaction Fraud": If predicting Fraud and include Support Ticket ID Filed for Fraud Recovery as a feature, the data leaks the final outcome.

Result: You should drop the column in the Training data.

ETL Pipelines

Processing steps

1. Extract = Retrieve raw data from sources (DBs, flat files…). Ensure data integrity. Real-Time or batches.

2. Transform = Convert raw data into suitable format. Data cleansing, enrichment, computations, encoding/decoding, format changes.

3. Load = Store transformed data into target (DWH, repo…). Ensure data maintains integrity. Batches or streaming.

Transformers and LLMs

Basic Concepts - Tokens and Embeddings

Tokens = numerical representations of words or parts of words
- A word can consist of 1+ tokens
- Punctuation signs (. “ ,) are also usually tokens
- Words/tokens can be loosely thought as the same, although strictly speaking they're obviously different
Embeddings = mathematical representations (vectors) that encode the “meaning” of a token

Evolution of the Transformer Architecture

1. RNNs and LSTMs

Recurrent Neural Networks (RNNs) are AI models designed for sequential data - like text or time series - by using internal memory to process inputs in order.

Long Short-Term Memory (LSTM) networks are a specialized, advanced type of RNN created to solve the "vanishing gradient" problem, allowing them to learn long-term dependencies that standard RNNs forget.

RNNs and LSTMs are obsolete with Transformers for many NLP tasks, though they remain relevant for time-series forecasting.

Feedback loop already present here
Useful for modeling sequential stuff like time series or language (sequence of words)
RNNs propagate the “hidden state” i.e. the previous output

2. Encoder-Decoder Architecture (e.g. for Machine Translation)
Encoders and Decoders are RNNs
Last Hidden State = huge vector that contains the meaning of the sentence “X0 X1 X2”
Decoder understands the huge vector and can translate back (outputs “Y0 Y1 Y2 Y3”)
Problem: the one vector tying encoder↔decoder creates an information bottleneck → Information from the start of the sequence may be lost

3. “Attention is all you need”
“Attention is all you need” = Revolutionary NLP paper from 2017
A hidden state for each step (word/token) instead of the whole sentence. Each word has weights for the other words in the sentence → “attention weights”. Represent how important the other words are for this specific word → context. thicker/bigger arrow in diagram = bigger weight.
Emerging concept for relationships between words
Deals better with differences in word order in the sentence
Lingering problem: RNNs sequential in nature → can’t parallelize it
Transformer-based LLM Diagram

4. (Modern) Transformer Architecture

Ditches RNNs for feed-forward neural networks (FFNNs)
Uses “self-attention” positional encoding → each word has position in the sentence encoded in it
Means we can process words in parallel since we don't lose the info of their position in the sentence!
Each word has attention weights from all other words embedded within it
The words with their self-attention weights are fed into FFNNs
Parallelizable → can train on much more data (whole Wikipedia, whole internet…)

Self-Attention

Self-Attention and Attention-Based Neural Networks
Each encoder/decoder has a list of vector embeddings for each token (representations of the meaning of each token)
Self-attention produces a new vector for each token. This vector is the weighted average of all token embeddings with respect to this particular token.

The “magic” is in computing the attention weights
The final vector captures the “meaning” of the token, with context
Meaning tied to the embeddings of the other tokens, and their context weights
Example: the word “novel” can mean either “book” or “original”, depending on context
Self-attention will capture different words in the sentence to determine what meaning “novel” has in that sentence.

Calculating Self-Attention

3 weight matrices learned through back-propagation: Query (Wq), Key (Wk), Value (Wv).
Every token gets a q, k and v vector by multiplying its embedding against these matrices.
Calculate a score for each token.
Scaled dot-product attention → score computed by multiplying (dot product) the query vector of a token with each key vector.
Other similarity functions can be used instead of dot product.
Softmax then applied to normalize scores.

(Optional) Masked Self-Attention
Mask prevents tokens from “peeking” into future tokens
Normally when we read, we also read in sequence: we know what we have read so far in the sentence, but not yet what comes later
GPT uses masked self-attention, but BERT does something else (masked language modeling)!!
Multiply value vectors with corresponding scores, then sum them up → Final z vector for token (final self-attention vector for a specific token)
entire process for each token → gets self-attention embeddings for each token
This can be done in parallel for each token!!
Self-attention embeddings can now feed the FFNN
Multi-Headed Self-Attention
q, k, v vectors reshaped into matrices → each row is a "head”
Heads can be processed in parallel

Applications of Transformers
Chat
But transformers by themselves are NOT good chatbots! Just the underlying building block!

Must be trained further to hold a conversation

Also need extra moderation wrapping (prevents them from e.g. generating offensive content…)

Question answering

Text classification (e.g., sentiment analysis)

Named entity recognition (NER)

Summarization
Not that different from Machine Translation: vector embeddings representing meaning of text can have a shorter or longer representation

Translation
Code generation
Text generation (e.g. automated customer service)
But deploy tools with care! Chatbots might agree to do stuff that never has a corresponding action. And people hate chatbots not understanding them.

GPT Architecture

Generative Pre-Trained Transformer (GPT)
Type of Large Language Models (LLMs), i.e. models that have been trained on a huge amount of human language data
OpenAI's GPT-2 is FOSS, although later models are closed
Other LLMs are similar to GPT-2

GPT is decoder-only
In contrast, BERT is encoder-only. Also, T5 is an example of a model that uses both encoders and decoders.

GPT has stacks of decoder blocks = masked self-attention layer + FFNN

GPT has no concept of input!!
All GPT does is continuously generate the next token in a sequence

Using attention to maintain relationships to previous tokens

GPT can be triggered to start (or “prompted”) with a sequence of tokens

It then keeps on generating given the previous tokens

Remember that the sequence can be processed in parallel! No need to feed in one token at a time.

Getting rid of the idea of inputs/outputs allows unsupervised training on unlabeled piles of text
GPT “learns a language” rather than being optimized for some specific task. Learns to interpret and speak it.

Hundreds of billions of parameters

LLMs

LLM Input Processing

Tokenization + token encoding (of prompt sequence)
Token embedding
Captures token similarities, i.e. semantic relationships between tokens, with vectors in a very high dimensional space.
Positional encoding
Captures token positioning in the input relative to other nearby tokens
Uses an interleaved sinusoidal function (both sine and cosine), allows it to work on any sequence length.
For a given period of e.g. 100 tokens, you can infer where the token is relative to its 100 neighbors.
For a really long sequence of tokens, this would eventually repeat, but by then the token is probably not relevant to the context any more.

LLM Output Processing
Stack of decoders outputs a vector at the end
Output vector contains the “meaning” of what we want to say
Multiply output vector with token embeddings
Result is logits (probabilities) of each token being the correct next token in the sequence
Final output can be randomized from logits (e.g. increase the “temperature”) instead of always picking the highest probability token → increases GPT’s “creativity”

LLM Key Terms & LLM Inference Parameters
  Top P = Threshold probability for token inclusion (higher = more random)
  Top K = Alternate mechanism where K candidates exist for token inclusion (higher = more random)
  Temperature = Level of randomness in selecting the next word in the output from those tokens
  High temperature → More random/creative
  Low temperature → More consistent
  Context window = Number of tokens an LLM can process at once
  Max tokens = Limit for total number of tokens (on input or output)

Transfer Learning (Fine Tuning) with Transformers

Use a pre-trained model as a base model, then adapt it (fine tune it) with your own data for your purposes.
Allows starting off with giant models (e.g. those that understand English, Klingon or Python) instead of having to train one from scratch!
Opens up a whole new world of AI applications.

Types of fine tuning
Add additional training data through the whole thing
Freeze specific layers, re-train others
E.g. train a new tokenizer to learn a new language → train just the tokenization steps
A popular technique here is LoRA (low-rank adaptation)

Add a layer on top of the pre-trained model
Just a few may be all that’s needed!

Can provide examples of prompts and desired completions
e.g. “How’s the weather?” → “What’s it to you, bucko?”
Can adapt it to classification or other tasks
e.g. “Wow, I love this course!” → Positive emotion

Hugging Face
  Giant repository of pre-trained models you can use → huggingface.co
  Can mess around with models… for free!
  Contains also a ton of learning resources
  Many available models: GPT-2, LLaMa, Stable Diffusion…

Decrease Training Time

by increasing the Batch Size and implementing Distributed Training.

Improve Fraud Detection

a) Use precision and recall to evaluate the model instead of accuracy

b) Adjust the decision threshold (default 50%) to improve recall

c) Perform a detailed analysis of false negatives

Training Restriction

Production inference data must always be transformed using the exact same parameters used in the training dataset. Ensures scaling math remains consistent with what the model learned during training.

Improve Run Time

Use speculative decoding to evaluate and accept the whole block of tokens in a single parallel pass rather than token by token.

ML Exam: 2 - Background