ML Exam Prep - Data Ingestion and Storage

ML Exam Prep 

Data Ingestion and Storage


Types of Structure of Data


Properties of Data (4 Vs)

1. Volume (Size) - GBs? PBs?…

2. Velocity - High velocity → Real-Time or near-RT processing

3. Variety - Structured? Mixed? Multiple sources? Multiple formats?

4. Veracity?


Data Warehouses, Data Lakes, Data Lakehouses

- Data Warehouse (DWH) (e.g. Amazon Redshift) - Centralized repository optimized for analysis (read-heavy operations) where data from different sources is stored in a structured format

- Data Lake (e.g. Amazon S3 can be used as data lake)  - Storage repository that holds vast amounts of raw data in its native format (predefined structure is not necessary). Structured, semi-structured, & unstructured data

- Often, organizations use a combination of both, ingesting raw data into a data lake and then processing and moving refined data into a data warehouse for analysis

- Data Lakehouse (e.g. AWS Lake Formation with S3 & Redshift Spectrum) - Hybrid data architecture, tries to provide advantages of both. Performance, reliability & capabilities of DWHs. Flexibility, scale & low-cost storage of data lakes.

Data Mesh

Domain-based data management paradigm. Decentralized architectural framework that shifts data ownership from a central team to domain-specific teams (e.g., marketing, sales, shipping).

ETL Pipelines

Processing steps

1. Extract = Retrieve raw data from sources (DBs, flat files…)
    - Ensure data integrity
    - Real-Time or batches
2. Transform = Convert raw data into suitable format
    - Data cleansing, enrichment, computations, encoding/decoding, format changes…
3. Load = Store transformed data into target (DWH, repo…)
    - Ensure data maintains integrity
    - Batches or streaming

Managing ETL pipelines

- Automation
    - AWS Glue
    - Orchestration services (EventBridge, Lambda…)

Data Sources

- JDBC (Java DB Connectivity) = Platform-independent, Language-dependent (only Java)
- ODBC (Open Database Connectivity) = Platform-dependent (thx to drivers), Language-independent
- Other: Raw logs, APIs, Streams…


Data Formats

  • CSV (Comma-Separated Values)

    • Human-readable & editable data storage
    • Small-medium datasets → import/export for DBs & spreadsheets
    • While CSV normally refers to commas (,), the underlying data format is the same no matter the separating character → character can also be a tab (TSV), blankspace...

  • JSON (JavaScript Object Notation)

    • Human-readable, Flexible schema with nested structures
    • Use cases: data exchange between web servers-clients or programming languages, configurations/settings for SW apps, RESTful APIs…
    • JSON Lines (JSONL) = newline-delimited JSON → subformat of JSON

      • Each JSON object is separated by a newline character

      • Advantage: individual lines/objects can be read/processed independently (without requiring entire dataset to be loaded into memory)

        • Allows efficient streaming for real-time applications
        • Ideal for large-scale data processing
        • Easy appending of new records

      • JSONL screenshot

  • Apache Avro

    • Row-based binary format, stores both data & its schema
    • Use cases: big data, real-time processing systems, schema evolution needed, efficient serialization for data transportation…
    • Used in Apache systems (Kafka, Spark, Flink), Hadoop ecosystem.
  • Protobuf (Protocol Buffers)
      • Originally designed by Google
      • Similar to Avro: row-based, binary, ideal for serialization, supports schema evolution…
      • Many SageMaker built-in algorithms expect this data format for their training data
        • Usually specified as “RecordIO-Protobuf”
  • Apache Parquet
    • Columnar storage format, optimized for analytics
    • Excellent data compression & encoding algorithms
      • reduced storage space
      • improved query performance
      • Use cases: analyze large datasets, read columns instead of rows/records, optimize storage & IO operations
      • Used in Apache systems (Spark, Hive, Impala), Hadoop ecosystem, Amazon Redshift Spectrum
  • Apache ORC (Optimized Row Columnar)
    • Columnar storage format (like Parquet)

Comments

Post a Comment

Popular posts from this blog

GHL Email Campaigns

GHL Email Campaigns Options  Option 1)    Tag free - Contacts > Quick Search box > filter data clicking to select > "Add to Automation" button > pick automation.  Option 2)   Automations > Workflows tab > pick workflow > start workflow with trigger component > click on it > pick "Contact Tag" >  "Add Filters" button > on filter add one that is "Tag Added" > pick which  type of tag. What Are The Tags 1) Settings > Tags > list of tags How to Add Tags 1) Bulk - Contacts > Quick Search box > filter data clicking to select > "Add Tag" button > pick tag 2) Single - Add tag when add contact 3) Bulk - ( Only Admin ) - In file to import, put Tags in column - Contacts > Import Contacts button > Contacts > decide columns. Everyone seems to tag the import with date AND tag it cold or hot if they can. 4) 
Read more

Whitelabel Options

Whitelabel Options   Whitelabel is selling someone else's solution but with your own label.  So different from car dealer where the cars still are known by the original manufacturer, but more like a generic product with your brand like Walmart's "Great Value" that does not manufacture the item.  Basically you are rewrapping someone else's solution. You become a non-employee salesman, but with your own company. 1) AiFrontDesk https://www.myaifrontdesk.com/white-label Stripe Rebilling Feature Gating Website Embedding    iframe on website    route their domain to our backend GTM Advice 24/7 Technical Support Detailed Analytics They want $100/license for first 10, then $50/license for next 40 so not economically viable if only $100/mth since only $37k/year. Would have to do $200/mth or more.  Their basic is $65/mth and $50/mth if paid year up front.  So they must prefer competitors. Paige at    1 (201) 228-1089 A better deal would be "M...
Read more

Free AI Tools

Free AI Tools Outbond: https://www.outbond.io/ Google AI Studio: https://aistudio.google.com/live    AI sees your screen - Excel expert Gamma: https://gamma.app/signup?r=mgp7dnjaverd8n2    visualize stuff - create sales decks  Hostinger: https://hostinger.com/wearenocode    website builder Suno: https://suno.com/    create song Leonardo AI: https://leonardo.ai/  Flow State AI  Eleven Labs: https://try.elevenlabs.io/mixajdyhxnx3  Realistic Voices.  Pauses with ...  Stress with capitalized letters. Runway AI: https://runwayml.com/   1. Animate Drawings: https://aurelius.link/meta-sketch 2. Text-to-Speech Generator: https://aurelius.link/text-to-speech-... 3. AI Headshot Generator: https://aurelius.link/headshot-generator 4. Documator: https://aurelius.link/documator 5. Free AI Transcriber: https://aurelius.link/riverside-trans... 6. Infographic Generator: https://infografix.app/ 7. Whybot: https://aurel...
Read more