Data Engineering Slack
Communities, Channels and Etiquette
Data engineering Slack communities are workspaces where data engineers ask questions and follow a tool or a course together: the DataTalks.Club Slack for courses and careers, along with project workspaces for Apache Airflow and Apache Iceberg and the Slack listed by Delta Lake. Apache Kafka and Apache Spark list no Slack, so check each project's community page for where it wants questions asked.
What are data engineering Slack communities?
A data engineering Slack community is a Slack workspace where practitioners ask questions and follow a tool or a course together. The ones confirmed on this page are the DataTalks.Club Slack, which is built around courses and careers, and the workspaces that Apache Airflow and Apache Iceberg point to from their own community pages, along with the Slack listed by Delta Lake.
Some of the projects data engineers rely on do not use Slack at all. Apache Kafka and Apache Spark list mailing lists and Stack Overflow tags on their community pages and no Slack workspace, so knowing which venue a project uses saves a lot of searching. Every name on this page was read on the community's own page, and no member counts appear because they change and the sources disagree.
Data engineering Slack workspaces and where other projects send you
The DataTalks.Club Slack: courses, careers and general channels
The DataTalks.Club Slack guide describes a workspace of data professionals with curated channels for courses and careers and for learning, plus meetups and social chat, and says the workspace holds more channels than the list shows. Browse the channel directory instead of relying on the list.
For data engineers the named entry point is #course-data-engineering, the channel for the Data Engineering Zoomcamp. The course repository calls it a free 9-week course on building production-ready data pipelines, and tells learners to use that channel for troubleshooting and networking, including self-paced learners, who are told to ask questions and share progress there. A certificate needs the final project and peer reviews during a live cohort. The general #engineering channel covers engineering discussion but is not specific to data engineering.
The guide also sets rules that matter before you post. The promotion channels (#events and #interesting-content together with #shameless-social and #shameless-promotion) require you to follow the shameless promotion rules and its template, and #jobs has its own guidelines. Read those first, since a post in the wrong channel is the quickest way to look careless.
Know Data Engineering Slack the way the interviewer who asks it knows it.
Project Slack workspaces: Airflow, Iceberg and Delta Lake
A project workspace puts you next to the people who maintain and run one tool. The Apache Airflow community page sends ad hoc questions to the official Airflow Slack first and names 5 channels: #user-troubleshooting for help using Airflow, #user-best-practices for asking about and sharing usage practices, #new-contributors for questions during a first contribution, #contributors for in-depth contribution discussion and #documentation for documentation improvements.
Apache Iceberg says it uses the Apache Iceberg workspace on Slack and gives an invite link on its community page, and Delta Lake lists Slack and Google Groups on its community page, along with GitHub. Take the invite from those pages directly: a workspace link copied from a third party list can be out of date, and the project page is where the current one lives.
Whichever workspace you join, the Airflow README notes that its code of conduct covers GitHub and the mailing lists as well as Slack and anywhere else the community meets, so the same standards apply in a channel as in a pull request.
Where projects want questions: Slack is 1 venue of several
Airflow's community page lists 3 places to ask a question: the Airflow Slack and Stack Overflow, plus the Users list. It points to Slack first for ad hoc questions, names the Users list as the preferred channel for important announcements such as releases and security and for asking other users for help, and uses the Dev list for announcements, as well as proposals and votes.
Iceberg works the same way: its community page says most discussion, including design proposals and votes, happens on the dev list, and the Slack workspace sits beside it. Kafka and Spark go further and list no Slack. Spark's community page points usage questions to the apache-spark Stack Overflow tag and the user mailing list, asks you not to cross-post between Stack Overflow and the lists, and says its Gitter room for Spark with Scala is not officially part of the project.
The practical rule is to read the project's community page before you pick a venue. A quick question belongs where the project says quick questions go, and a design proposal belongs on the list where the project holds its votes.
How to join a data engineering Slack and its channels
- 01
Find the invite on the community's own page
Open the project or course page, such as the Airflow and Iceberg pages or the DataTalks.Club page, and follow the Slack link there.
- 02
Open the channel directory
On desktop, go to Home, then Directories, then the Channels tab; on mobile, search by name from Home. Public channels can be searched by name or description.
- 03
Join the channels you will use
Select a channel and choose Join Channel. Private channels need an existing member to add you, and guests cannot browse or join public channels.
- ▸On desktop, the arrow beside Join Channel lets you join for a day or 48 hours, or for a week, and be removed automatically
- 04
Read the channel's rules before posting
Promotion channels and jobs or help channels each have their own norms, so check the pinned guidance or the community's guide before your first message.
What a data engineering Slack is good for and where it stops
Slack suits asking while you work, and the Zoomcamp channel does that for each cohort. It also puts you in reach of people who run the same tool, which is what Airflow's help channels are for. And it brings together practitioners across employers, which the DataTalks.Club career and meetup channels aim at.
The pages also show the limits. Slack is not where open source projects hold their decisions: Airflow and Iceberg both keep proposals and votes on the dev list. Not every project has a workspace at all, and Kafka and Spark list none. Access varies as well: private channels need a member to add you, and guests cannot browse public channels. A workspace can hold more channels than its guide lists, so treat the guide as a starting point.
Treat a Slack answer as a lead to confirm in the documentation, and write down what you learn. A fix someone gave you in a thread is only as good as the version and setup it was given for.
Before you post a question in a data engineering Slack
- Search the channel and the project's documentation for your error message first
- Post in the channel that matches the question, such as a troubleshooting channel for a failure
- Include the tool version and what you ran, then the exact error and what you expected instead
- Put long logs or code in a snippet or gist and quote only the few lines that matter
- Keep promotion and job posts to the channels the community sets aside for them
- Do not post the same question in several venues at once
From Slack threads to interview practice
The threads in a help channel are a steady supply of pipeline failures described by symptom: a task that hangs, a load that comes in short, a schema that changed. Reading how others diagnose them trains the question an interviewer asks in the data engineering system design round, which is how you would know a pipeline is broken and find the cause.
When you want that under interview conditions, work through data pipeline interview questions and the data pipeline practice problems that ask you to design and debug the pipelines these channels talk about. If you run Airflow, Airflow interview questions covers what interviewers ask about it.
Data engineering Slack FAQ
What are the best Slack communities for data engineers?+
How do I join a data engineering Slack community?+
Which Slack channel should I use for a data engineering course?+
Where do I ask an Apache Airflow question?+
Do Apache Kafka and Apache Spark have a Slack?+
Is Slack the right place to propose a change to an open source project?+
The candidate who gets the offer
- 01
Reading a solution is not the same as writing one
Every engineer who has frozen on a query they had read a dozen times knows the gap. The only preparation that closes it is producing the answer yourself, under time, before the interview does it for you
- 02
76% of hiring managers reject on the coding task, not the resume
From HackerRank's 2024 Developer Skills Report. Candidates who look strong on paper still fail the live screen if they haven't done timed, executable practice
- 03
System design comes down to the calls you defend out loud
Ingestion, batch vs streaming, the bronze/silver/gold layers, idempotency, backfill and replay. Sketching the pipeline and naming the failure modes is the signal, not the boxes
Related guides
Data pipeline interview questions
The pipeline design and operations questions interviewers ask
Data pipeline practice problems
Pipeline design problems to practise before the loop
Data engineering interview prep
The rounds and questions, plus a plan for the loop
Data engineering community
Where to join and ask beyond Slack
Data engineering newsletters
Weekly reading that pairs with a Slack workspace
Data engineering Discord servers
Servers for engineers who prefer Discord
Compete in the Weekly
Dirty, production-shaped data, scored blind each week.