Why Every AI Project Depends on Data Infrastructure
Every AI use case described in this blog — demand forecasting, lead scoring, HR analytics, fraud detection — depends on the same foundation: data that is accessible, consistent, timely, and governed. Without that foundation, AI projects fail not because the algorithms are wrong but because the data they run on is wrong.
Data pipeline automation is the unglamorous prerequisite for every AI deployment. It is also the area where the largest gap exists between organizations that successfully run AI in production and those that build impressive demos that never scale.
This post covers the technical architecture choices that determine whether AI data infrastructure works in production, with specific reference to the Canadian regulatory and organizational contexts in finance, government, and healthcare.
ETL vs ELT vs Data Mesh: Choosing the Right Architecture
The choice of pipeline architecture is not primarily a technology choice — it is an organizational and use-case fit choice.
ETL (Extract, Transform, Load) applies transformations before data lands in the target system. This approach is well-suited to: organizations with limited cloud data warehouse compute budgets, data from a small number of well-structured sources, and use cases where the transformation logic is stable and unlikely to change frequently. ETL's weakness is inflexibility: changing the transformation logic requires reprocessing historical data.
ELT (Extract, Load, Transform) loads raw data into a cloud data warehouse and applies transformations using warehouse compute (BigQuery, Snowflake, Redshift, or Databricks). This is the dominant architecture for modern data teams because: raw data is preserved for re-transformation when business definitions change; transformation logic is version-controlled in SQL or dbt; and cloud warehouse compute is elastic, so heavy transformation runs don't require dedicated infrastructure.
Data pipeline via data mesh is appropriate when: an organization has multiple distinct business domains generating data (e.g., a large financial institution with retail banking, commercial banking, and wealth management divisions), each domain's data has domain-specific transformation logic that the central team cannot effectively own, and the organization's data team is large enough to support domain-embedded data engineers. Data mesh addresses the bottleneck that appears when a central data engineering team becomes responsible for understanding and transforming data from dozens of source systems they did not build.
Data Quality: The Constraint That Determines AI Outcome
The most common reason AI projects fail in production is data quality, not algorithm choice. A demand forecasting model trained on inventory records with 15% missing values will produce systematically underestimated forecasts. A credit risk model trained on data where three different systems represent the same customer entity with three different ID formats will generate predictions contaminated by entity resolution errors.
Effective data quality management in AI pipelines involves:
Schema validation at ingestion: Every record entering the pipeline is validated against a defined schema before processing. Records that fail schema validation are quarantined and logged — they do not propagate corrupt data downstream silently.
Referential integrity checks: Relationships between entities (customer IDs in transaction tables that do not exist in the customer master, for example) are detected and flagged before analytics or model training consume the data.
Statistical monitoring: Key data distributions — the average invoice amount, the proportion of null values in critical fields, the distribution of transaction timestamps — are monitored continuously. Deviations from expected distributions trigger alerts, catching upstream system changes before they corrupt downstream analytics.
Data lineage tracking: Every data asset has documented lineage — what source systems it was derived from, what transformations were applied, and what downstream dashboards and models depend on it. When a downstream output is questioned, lineage documentation provides the debugging path.
For Canadian government and healthcare organizations subject to the Privacy Act, provincial health information legislation, or PIPEDA, data pipelines must also enforce data minimization (only collecting and retaining personal information necessary for the defined purpose) and data retention controls (automatically purging records that have exceeded defined retention periods).
Streaming vs Batch: Matching Infrastructure to Decision Cadence
The choice between streaming and batch data pipelines should be driven by the decision cadence it supports — not by what is technically interesting.
Streaming pipelines process data as it is generated and are required for: real-time fraud detection in payment processing (where a decision must be made in sub-second timeframes before a transaction is approved), clinical alerting systems that must notify care teams of deteriorating patient vitals within minutes, and operational dashboards that must reflect current system state for active monitoring.
Batch pipelines are adequate and significantly cheaper for: financial reporting that is produced daily or weekly, operational KPI dashboards reviewed in weekly or monthly management cycles, and model training pipelines that run on a defined schedule rather than continuously.
The data insights analytics layer that consumes pipeline output should determine the architecture, not the other way around. A batch pipeline supporting a daily executive dashboard is a correct and cost-efficient architecture. A streaming pipeline supporting a live fraud alerting system is also correct. Streaming for a use case that only requires daily data is engineering waste.
Feature Stores: Enabling AI at Organizational Scale
Organizations deploying more than a handful of AI models encounter a specific infrastructure problem: each data science team computes the same derived data attributes independently. Customer tenure (calculated from account open date and current date), trailing 90-day transaction volume, and geographic risk segment are used across dozens of models — and each team computes them slightly differently.
This inconsistency creates two problems: training-serving skew (the feature is computed differently during model training than during production serving, causing the model to behave differently in production than in testing) and feature duplication (engineering resources are spent recomputing the same features rather than building new ones).
A feature store solves both problems. Computed features are registered once, made available to all models through a consistent API, and served with point-in-time correct historical values for model training. Teams building new models search the feature store for existing features before computing new ones — dramatically reducing the engineering cost of new model development.
For Canadian financial institutions under OSFI Model Risk Management guidelines, a feature store also provides the feature computation documentation required for model validation: what each feature represents, how it is computed, what data it derives from, and what models use it.
Related reading: AI for finance teams covers how financial reporting pipelines integrate with accounting systems to produce audit-ready data assets.