Tuesday, June 10 — Interview Insights, Docker Recap

🧠 Learnings from Interview -

  1. What all did you optimize at work?
  2. Medallion Architecture
  3. How much data do you process every schedule?
  4. What kind of preprocessing/cleaning/feature engineering do you do?
  5. How do you handle failures in a data pipeline?
  6. New Task at Work – Kinaxis CSV Files via SnowSQL
  7. Shift from Previous Company
  8. Data Cleaning
  9. Snowpark
  10. Helped Set Up Conjur for Tableau

To Learn:

  • loading Kinaxis-generated CSVs using snowsql.
  • Add validation + notification on load failure.
  • scheduling via Snowflake Tasks and parameterized Stored Procedures
  • Apache Airflow for orchestration — retries, alerts, on-failure callbacks.
  • External monitoring tools like CloudWatch, PagerDuty, and Slack alerts.
  • idempotent task design and partial failure handling in DAGs.
  • Snowpark for Python for writing modular transformation logic.
  • Snowpark Especially useful when working with ML models inside Snowflake Cortex.
  • Data Cleaning - handling null values, outlier detection, and schema drift.
  • feature engineering feature normalization, encoding, and column-level lineage tracking. -Catalyst Network - Policy Enforcement Pipeline
  • Data Cleaning - Great Expectations
  • how I helped configure Conjur (Secrets Manager) to securely inject database credentials into Tableau Server. Used it to rotate Snowflake passwords without redeploying dashboards or scheduled extracts.
  • the tech stack transition: from mostly on-prem Spark/Hadoop to cloud-first Snowflake, dbt, Airflow setup.
  • Also a shift in role: From ETL builder to a more platform + orchestration + analytics hybrid.
  • exploration of GenAI for unstructured data.
  • trying out Snowflake Arctic and studying agentic AI patterns via LangGraph.
  • Medallion : structure and clean data in stages, improving quality and lineage tracking.

🚀 Finished Docker Labs today:

  • Towards the end, was just searching all the answers online. So need to be more thorough on Docker Compose, Registry, Networking and Storage.
  • My Docker Notes via Gemini

Reflection is the bridge between learning and wisdom. 🔁