ML Exam Prep: 8 - Model Fitting

 ML Exam Prep

Model Fitting

1. Overfitting (Model is too complex)

The Problem: Memorizes the training data (including its noise and random errors) instead of patterns. It scores 99% on training but performs terribly on new data.

How to Prevent It:

• Increase Regularization: Penalizes large weights; L1 (Lasso) zeroes out weak features, L2 (Ridge) shrinks them.
• Increase Dropout: Randomly shuts off neurons during training to force generalized learning.
• Fewer Feature Combinations: Removes complex or noisy inputs to stop hyper-specific conclusions.
• Early Stopping: Halts training the moment validation loss begins to rise.
• Data Augmentation: Tweaks existing training samples (e.g., rotating images) to create new variety.

2. Underfitting (Model is too simple)

The Problem: Fails capture the basic data patterns, performing poorly on both training and testing sets.

How to Prevent It:

• Increase Model Complexity: Add layers/neurons or switch to a stronger algorithm.
• Decrease Regularization: Reduce dropout or L1/L2 constraints so the model is free to learn.
• Increase Feature Combinations (Feature Engineering): Combine or expand inputs to reveal complex relationships. Ex: combining width and length into area.
• Train for More Epochs: Give the model more training time to find the global minimum.

3. Catastrophic Forgetting (Model forgets old tasks)

The Problem:  In sequential learning, training on a new task completely overwrites the weights learned from previous tasks. Ex: trained on find cats and then trained find cars, it completely overwrites its weights and entirely forgets how to do find cats. 

How to Prevent It:

• Regularization-Based (Elastic Weight Consolidation - EWC): : Identifies weights critical to old tasks and penalizes changing them.  
• Rehearsal / Interleaving: Mixes a small amount of old task data into the new task training set.
• Architecture-Based (Progressive Neural Networks): Freezes old network columns and adds new neurons for new tasks.
•