Curated, analysis-ready market datasets that power downstream forecasting and analytics · 2016 · Python data engineering
Role at the time: Data Engineer · (2016 — Data Engineer → Senior Data Engineer years) Type: Portfolio case study — architecture & approach are representative; production code is proprietary.
Portfolio managers, data analysts and quantitative researchers wanted to study market trends and relative asset-class performance — but the raw material from Morningstar, Bloomberg, FactSet, LIM and LANA arrived in inconsistent formats, identifiers and frequencies. Every analyst was re-cleaning the same vendor feeds before they could do any real analysis.
This project (circa 2016) is a data engineering build: I ingested, cleaned and conformed those feeds into consistent, time-indexed, analysis-ready datasets — a lightweight feature platform — so the downstream consumers could run their own EDA, forecasting (ARIMA/LSTM) and ML signal models (Random Forest/XGBoost) without ever touching raw vendor data. I built the data substrate; the modeling was theirs. It marks the stage where my pipelines began explicitly serving data scientists and analysts as first-class customers.
flowchart LR
SRC[Market data sources<br/>Morningstar · Bloomberg · FactSet · LIM · LANA] --> ACQ[Acquisition<br/>APIs + web scraping]
ACQ --> CLEAN[Cleaning &<br/>standardization]
CLEAN --> CONF[Conform to<br/>entity × date × metric panels]
CONF --> FEAT[(Feature tables<br/>returns · rolling stats · macro)]
FEAT --> AN[Data analysts<br/>EDA · dashboards]
FEAT --> DS[Data scientists<br/>ARIMA · LSTM · XGBoost]
- Languages: Python, SQL
- Ingestion: vendor APIs, web scraping
- Data wrangling / pipeline: pandas, NumPy
- Storage / modeling: relational store + materialized feature tables
- Serving to consumers: clean time-indexed panels and engineered feature tables
- (Downstream, by analysts/scientists, not me: ARIMA, LSTM, Random Forest, XGBoost)
- Conformed time panels — heterogeneous vendor feeds standardized into consistent
entity × date × metricframes so any consumer joins them the same way. - Lightweight feature store — reusable engineered features (returns, rolling statistics, momentum, macro indicators) materialized once and shared, instead of every analyst recomputing them.
- Point-in-time correctness — features stored with as-of timestamps so downstream backtests don't suffer look-ahead leakage. The pipeline guarantees the temporal integrity; the modelers rely on it.
- Build the substrate, not the model — my deliverable is clean, trustworthy, reusable data; EDA and forecasting are downstream workloads I enable.
- Conform once, consume many times — standardizing vendor feeds centrally removed duplicated cleaning effort across the analytics team.
- Engineer features as a shared asset — a common feature layer gives every consumer the same definitions and prevents metric drift between analysts.
- Bake in point-in-time discipline — as-of timestamps protect downstream backtests from leakage at the data layer, where it belongs.
- Self-service analytics — analysts and data scientists consumed ready-to-use datasets instead of wrangling raw vendor feeds.
- Consistency across the team — one conformed schema and shared features eliminated divergent, ad-hoc cleaning.
- Faster time-to-insight — downstream forecasting/ML work started from clean features, not from scratch.
- Established the feature-engineering and consumer-serving patterns that later scaled onto the lakehouse.
Part of my journey toward Data & AI Platform Engineer — the 2016 stage, when I was a Data Engineer and my pipelines began explicitly serving downstream analysts and data scientists.
⏮ prev: structured-finance-pricing-pipeline-python-sql · ⏭ next: predictive-error-rca-ml-nlp Full journey: https://kamalakarpeta.github.io