- The author joined Deepnote without a data background but found the field adjacent to software engineering.
- Realizing a lack of knowledge about data tools and workflows, they learned through articles, colleagues, and client interactions.
- The article is a guide for software engineers to understand data teams, covering data lifecycle, tools, and professions.
- Data professions include analytical, scientific, engineering, and machine learning types, each with distinct roles and tools.
- Data lifecycle involves ETL/ELT processes: extracting, transforming, and loading data from various sources.
- Data storage includes file formats (CSV, Parquet), memory formats (Arrow), and systems like warehouses, lakes, and lakehouses.
- Processing tools range from SQL and local DataFrames (pandas) to distributed systems (Spark) and stream processing (Kafka, Flink).
- Orchestration (e.g., Airflow) and observability tools manage pipelines and data quality.
- Data consumption includes dashboards, operational analytics, ad-hoc analysis, and embedded analytics, with tools like Tableau and Metabase.
- The article emphasizes data governance and concludes with an overview of how data flows from source to use.