Data Engineering Community

Start from the problem, not the platform. If a job fails inside a tool, go to that tool's own channel, because the people there maintain it or run it daily. Apache Airflow lists Slack channels for troubleshooting and for how to run it well, and has GitHub Discussions too, while Apache Iceberg links its Slack from its community page. If the question is about design or a career move, a broad community is better, such as the DataTalks.Club Slack or the dbt Community, where people have seen several teams' versions of the same problem.

Broad communities also differ in who they serve. The dbt Community centres on dbt and analytics engineering. Locally Optimistic describes itself as a Slack group for current and aspiring analytics leaders, and you join by emailing a request with a short introduction, so it fits an engineer who works close to analytics more than one who builds streaming systems. The r/dataengineering subreddit is the widest of the group, and its monthly digest and the Data Engineering Wiki are the community's own attempts to keep what it learns in one place.

For public sector work, the UK government runs a cross-government data engineering community with a mailing list and a Slack channel. Its page says you don't need to be a data engineer to join, and that some events are open to all public sector staff.

Chat is fast and forgets. A mailing list or a Stack Overflow tag is slower and keeps its answers where a search engine can find them, which is why the Apache projects still send users there. The Apache Spark community page splits the 2 jobs: usage questions, such as how to use an API, go to the apache-spark tag on Stack Overflow, while broad questions that invite opinion, and debugging, go to the user mailing list. The same page keeps the developer list for people who want to contribute code.

Apache Kafka's contact page follows the same pattern, with a user list for general questions and a developer list for development discussion. The practical rule: search the archive before you ask, because the answer to a first-time question is usually already there, and write your question so it still makes sense to a stranger a year from now.

A course gives a community something to talk about. The Data Engineering Zoomcamp from DataTalks.Club says all of its course content is free and open source, and it runs in 2 modes: a live cohort with deadlines and a leaderboard, and a self-paced mode you can start at any time. Its tools list includes Docker, PostgreSQL, Terraform, Google Cloud, BigQuery and dbt, and also Apache Spark with Kafka. Its community channel is #course-data-engineering on the DataTalks.Club Slack.

The value of a cohort is that everyone hits the same wall in the same week. A question that took you an evening has usually been asked in the channel already, and answering someone else's question is the fastest way to find out whether you understand the step. If you finish a course and want something of your own to show, data engineering projects are the natural next step, and a data engineering hackathon gives you a deadline and teammates.

Events are where a community meets in person. Some are run by a project's users and some by a vendor, and the difference matters when you read a schedule. Airflow Summit is a conference on Airflow and data orchestration, run by an events company, and its site says the Apache Software Foundation isn't affiliated with it. It also offers on-demand recordings and a free online reconnect, so you can follow it without travelling. Spark's community page notes that Spark meetups are grass-roots events organised by individuals around the world, and the dbt Community lists meetup groups in cities from New York to Lagos to Berlin.

Vendor conferences, such as Databricks' Data + AI Summit and AWS re:Invent, along with Snowflake Summit and Google Cloud Next, are run by the vendors themselves. They are strongest when a session covers a tool you run, and the useful talks are usually the ones given by engineers from other companies about what broke. Dates and locations change every year, and so do ticket prices, so take them from the organiser's own site.

Reading is a community too, only quieter. Data Engineering Weekly says it reviews more than 15 articles and papers each week and picks a few of them; it aims to stay vendor neutral and labels sponsored content. The Data Engineering Podcast, hosted by Tobias Macey, describes itself as going behind the scenes for the tools and techniques of the field, and its difficulties.

These sources tell you what is changing, while a community lets you ask about your own situation. Pick 1 digest for breadth and 1 voice you trust for depth, and see the best data engineering newsletters for how to choose between them.

A community cannot take the interview for you. People post the problems they were asked and how they reasoned through them, so you hear the follow-up questions, not only the first one. You can describe a pipeline design out loud in a thread and find where others would push back. And the people who run a tool daily can tell you which of its features come up in real work, which is what an interviewer is probing when they ask about it.

Reading is not enough on its own. The round still asks you to write the query or draw the design under time, so pair the community with timed practice. The data pipeline interview questions and data pipeline practice problems cover the design round, Airflow interview questions cover the orchestration questions that come up most often, and the data engineer interview prep guide lays out the rounds and how to plan them.

DataDriven runs a Weekly, one community data engineering challenge a week. Each comes with a brief, a small sample dataset and a starter file, and your solution is scored blind on hidden data, so you cannot tune it to the answer you can see. Everyone works the same problem in the same week, which gives you a shared result to compare notes on.

The discussion forum is where members talk through problems and tools, or career questions, with other data engineers.