Data

My Data Journey

Started September 27th 2024 whilst attempting to learn Technology:Databricks

I want to document my journey here.

My Data Journal

2024

September

• (27) I imported 600MB of json inventory data into Technology:Databricks as my first poc. I learned that my data was incomplete 😱

October

• (3) Trying to find a good starting point, I landed on https://www.reddit.com/r/dataengineering/ which lead me to https://dataengineering.wiki/Learning+Resources which lead me to https://dezoomcamp.streamlit.app/. I'm currently on Module 2 https://dezoomcamp.streamlit.app/Module%202%20Workflow%20Orchestration
  • I need to investigate what exactly the following things are, and how to use them (or if I want to use them)
    • airtable
    • maige (orchestration)
    • streamlit.app
  • I'm wondering if this might be a better way to keep notes https://quartz.jzhao.xyz/
  • I don't know where the Maige stuff is going to take me, I understand orchestration concepts I think I'll be better off
  • sticking to spark https://pages.databricks.com/rs/094-YMS-629/images/LearningSpark2.0.pdf