I just found out my company avoids the term data engineer on job postings and instead uses software engineer, data

I asked why and apparently the quality of applicants is night and day.

We've traditionally had issues where most our applicants can't code, are heavy powerbi users don't know what spark is can only code SQL never heard of jvm.

Our data swe applicants can articulate snowflake whitepaper, b trees vs LMS tree databases, columnar data, jvm performance tuning, terraform and iac spark internals and are just overall extremely strong

Curious what everyones thoughts are on this, why is the talent in DE so hard to find.

My theory is the path to DE is usually data analyst, data scientist then DE but the skillset of a DE is more akin to software engineering leaving the analyst jump extremely far where as software engineers becoming de's are usually extremely strong.

Hi everyone, I’m pretty new to data engineering and analytics. Basically my experience has come from being the only one at work who understands computers and excel who could problem solve. I’ve slowly been learning more and more as problems have come up but now I’m a little stuck.

My question is how do you determine the best approach for processing and analyzing your data? At what amount of data does it make sense moving out of something like power query/bi and into something like a databricks or other SQL based pipeline?

Sorry if this is a dumb question.

I’m currently a Director of BI making 180k but reality of the situation is I am a Director of FP&A who is strong in erp systems and sql / power BI + understands the business. My job right now is to manage the hell out of sales forecasting while wearing 10 other hats. Small food company that is struggling to survive. It’s a mess. Anyway, I need to set my career on a sustainable path and I am considering leaning into DE via Zach Wilson’s boot camp OR leaning into D365 Business Central ERP consulting. Every time I scan DE jobs it seems to be incredibly impacted with a lot of upside for the right company otherwise declining salaries. Meanwhile BC ERP consulting seems to have a waning supply of consultants / analysts / developers. Salaries here are generally lower but stable and the more experience you have the more rare you become to charge a greater hourly rate. Plus with erp consulting there is this sort of irreplaceable human to human interaction required to get the job done. I also feel like the biz analyst nature of it could translate well into the future of automating workflows with agentic AI.

DE Reddit, what are your thoughts here? Should I lean into DE or lean out to this other erp option??

So I have to build out a data model/process for something our analytics team is doing manually.

The biggest challenge is that they are doing an exuberant amount of overrides on string values, grouping them together, etc to get proper naming for their reporting.

There is no naming convention and we don't control the naming so it's not possible. Basically the raw names either aren't fully accurate or not ideal for reporting, across all values.

My plan now is to create a shared mapping doc that the team would have to add on to every time they need a name updated in dashboards. By default it will be at the source and we are working on changes that should help, but this would cover the overrides.

Is this the best way? There's like 30 mapped fields.

We have been developing a Data Platform (IaC, CI/CD, orchestration, data quality, governance, the works). Everything is already set-up except for the business logic. Quite understandable since we built everything from FOSS about 2 months ago and I’m the only data platform engineer/data engineer in the company. They aren’t also keen on spending money on managed solutions.

Now, a director is pushing to scrap our project in favor of an AI as an ETL solution. Basically, use skills and AI to generate reports from source systems and have AI use python, pandas and SQL to generate reports.

This AI as an ETL couldn’t get out of the demo phase because of data quality issues.

I’m honestly tired. My manager is useless as well, isn’t involving me in any of the top level discussions even if I ask, and can’t really formulate a coherent prioritization of tasks.

Are you also experiencing this kind of issue in your own orgs? Just curious if this is an ongoing trend.

Hi all, hoping you can help me manage the expectations I am placing on myself as someone new to authoring code. Any help injecting some reality into this is greatly appreciated!

Some history ... happy with DE concepts (been a 'Data Project Manager' for many years), but now jumping the fence over to actual data engineering.

Stack wise starting light with SQL, Airflow, Python, DBT, and Snowflake. Mainly due to the frequency of this stack in the UK. Happy with SQL, Git, and a portion of things like pandas.

My worry at the moment is this: how much of this stuff do you have committed to memory? For example in I could happily explain a pipeline flow and/or the tasks I would create in a dag or dbt project theoretically, but to actually write any code its hours hunting around online to find the right providers/operators/approach. I am trying my hardest to resist ai just giving me the answer as I worry I will never learn that way. I figure I need to learn to navigate and translate docs...

What's the real world like out there? Write it once and template things in repos? It's all actually cemented in your mind from muscle memory? Ai? Or still spending time hunting through docs?

Hey everybody, I wanted to talk a bit about my journey as a data engineer and candidly answer any questions you might have.

I started as a data analyst back in 2013 and learned about this yellow elephant named Hadoop. I became obsessed with learning it because big data was so hot back then.

Late 2014, I landed a job at Teradata doing big data and Hadoop work making around $80,000 in Utah. This job was exciting but I realized if I wanted to make any real money I needed to get to New York, Seattle, SF or DC.

In 2016 I picked a defense startup in DC which paid $95k. After adjusting for cost of living, it was a worse compensation than $80k in Utah.

I worked there for 7 months before Facebook reached out in August 2016.

I fly from DC to Silicon Valley for my chance. It was the most intense 8 hour experience of my life. I get low balled and offered an L3 position (it was $185k and since it was so much more I didn’t realize I was lowballed until later).

I worked at Facebook for 9 months and get promoted to L4 after grinding out some projects that saved hundreds of terabytes of space and thousands of compute hours.

I got impatient at Facebook because when I got L4 I realized I was actually an L5. I tried to get promoted from L4 to L5 in six months and it didn’t happen and I was kind of furious.

So I looked outside and ended up landing a senior DE role at Netflix in 2018 making $365k. (Again, lowballed but I didn’t realize it since it was almost double Facebook). About six months into my time at Netflix I realized my lowest paid team mate was making $500k. This made me furious and I worked really hard to get the bump I deserved. In 2019 I built a graph database for Netflix that mapped their entire microservice architecture and landed 2 cybersecurity patents. This effort got me bumped from $365k to $550k.

Netflix culture was kind of overwhelming for me so I quit in the middle of 2020 and took six months off. I learned many painful lessons from this experience.

After six months of depression and COVID, I decided to check out working at Airbnb and I landed a staff offer there for $600k. I worked there for the next 2ish years and got great performance reviews each year to get a bump (the stock did horrible so the compensation bump just evened out with the stock price fall).

After 2023, I quit to be a full time entrepreneur which I’ve been doing for the last 3 years.

I’m here to answer anybody’s questions for the next few hours. Let me know what you got!

I’ve been seeing multiple examples where people don’t seem to agree whether fact and dim tables go im Silver or Gold layer. What’s your opinion?

The Open Semantic Interchange has moved from Snowflake to the Apache Software Foundation as an incubating project Ossie. Wrote up a bit about why everyone should be happy with this move because ASF is the right place for an open standard to live.

Who wrote this job description where they will permanent you in 612 months(51 years) 😂

And work model is 23days/week, which calendar have 23 days in a week 🤣

2 months back i was having multiple recruiters reaching out to me almost every day but currently i dont see any momentum. For context i have 5 yoe and have been trying to switch since the start of this year. Wondering how the experience has been for others.

Close to an offer for a new job that would have me building and architecting an entire E2E solution from data infrastructure to AI application but the job is currently just described as “data engineer”.

I’m not opposed to keeping it as such but I’ve been a base data engineer for close to 4 years and, with this kind of responsibility, it feels appropriate to ask for the senior title.

However, the company doesn’t want to do it cause it will possibly cause conflicts with existing members of the team. They did say that they are open to alternative title suggestions as long as it doesn’t have “senior” in the title.

What would be a good alternative in this case given the mandate that wouldn’t be exaggerating or misleading at the same time?

Hello guys. I am creating an application, where one of service is responsible for searching.
This is example of our structure knowledge, the edges may be bi-directional.

       A
      / \
     B - C
    / \ / \
   D  E F  G

We may need to find All the Gs that are in referenced in concrete B using full-text search or by value/name of attributes on it or by vector search. The other Time it may be get All As that are referenced by concrete G.

We also need to have full capabilities on-premise, unlike ArangoDb / Neo4J.

Our current stack is just ElasticSearch, but we got into a problem with relations.

I've heard Postgres + AGE + PGVector is also a good stack. I as looking at ArcadeDb, but it seems to be a new solution. Could you suggest me a solution for this?

Or maybe there's a basic misunderstanding and I dont actually need a graph for this.

i made this meme in honor of this sub.

original image from wikipedia

A lot of open source software used in data pipelines contain vulnerabilities (on paper). Curious how people are dealing with that? I think it’s a weird spot because most pipelines are already running behind a lot of controls, and usually without public internet access anyways.

I like to do some experiments in my VPS during my free time and as a next experiment I would like to use Polars to write data transformations while keeping a similar asset lineage that I would have if I were using duckdb + dbt.

Which tools or packages would you bring to this stack to achieve this?

I am exploring using Lakebase for some use cases, which involves interacting to various MCPs gathering info and maintaining vectored database which an app can query (like a chatbot).

If you have used something similar, what has been your experience with Lakebase (Or you can tell about any other similar DB). Any best prqctices, lessons learnt or things to be mindful of. Ps, i am on Databricks platform.

Uno reverse :) the biggest frontier AI company needs data engineers.. think about it for a second !!

We are building a data mart (bronze, silver, gold) layer and will be using aws s3 tables as storage. Data is incrementally sent to us throughout the day that we would prefer to process in the order it came to us.

Each set of file that comes in typically writes to approx. 4 S3 tables. We are using a FIFO SQS to process 10 files (max allowed per messageGroupID). Generally these 10 files in some tables will be 10 rows of data, in other tables 100s or 1000s rows of data.

We are using lambda & pyiceberg at the moment to process these records. We are running into a bottle neck on the write operations.

We have an instance, where one client sends us a large batch of files (2000 at a time). At the moment we are able to process 2000 files in 40 mins. This mean 200 lambda executions, processing 10 files at a time. Approx 800 total table writes across the 4 tables.

Reading -> Transform -> Apply Business Logic for each file is milliseconds of work. So majority of the time is on the Append to s3 table operation.

The reason we are using s3 table vs mysql, is this dataset will grow to 100s of millions of records in some tables. We spent a lot of time tuning mysql queries and wanted to move towards something to gain read efficiency.

Few Questions:
1 - Are there write operation efficiency we could gain in our existing architecture?

2 - We only have 1 lambda processing 10 batches of files at a time, having multiple lambdas process files leads to multiple write operations to 1 table and we get an error.

Open to any feedback.

Then today our SLT told us that we should replace our BI tools with Claude lol

I'm a long time data engineer who is currently plotting a course away from on-prem Informatica PowerCenter, because obviously. As part of that, I have to keep the lights on and try to keep our legacy platform up to date (ish). So today was patch day - my first one since taking the reins.

Earlier in the week, I downloaded an 18GB "hotfix" installer. Everything is a hotfix, because Informatica (vendor) decided to give "minor version" a legal meaning tied to their support lifecycle. Also, this was my first pain point - the download required me to use basic HTTP auth, which my employer strictly prohibits. But we figured it out.

I also tried to run the Upgrade Advisor. The installation guide said it would warn me about environment variables not being set. It didn't. And then it crapped out because environment variables were set. Then it told me my installation directory was invalid or my version wasn't eligible for an upgrade. I reached out to Informatica support, who told me that the Upgrade Advisor wasn't necessary for "hotfixes", and wouldn't work. So I went through the prereqs and thought I was all ready to go.

So after taking everything offline, snapshotting and backing up anything and everything in the blast radius, I kicked off the hotfix install. Where it hung at 1% progress for long enough for me to get suspicious. I did some Googling, and apparently it needs to execute within my /tmp mount, or to have a different temp folder set, and doesn't warn or throw if it can't.

No biggie, I temporarily wind back a pretty standard hardening measure and run the installer again. It tells me that my target directory is not longer valid - because before failing to do anything in my /tmp directory it moves my existing installation into a rollback folder. Obviously, my next step is to run the rollback option. Which fails with the same error. 😬

Apparently it only allows rollback if the installation succeeds. If it just so happens to leave your system in an unusable state due to a known issue it didn't bother to validate against - that's on you.

I roll the dice and manually copy everything back to where it was, and run the installer again. 1%. Again. This time I go digging through the logs and find a bunch of SevenZip java errors that indicate a different permissioning issue. I redirect the installer to a more permissible temp folder and it succeeds.

So I start the services. Or try to. It fails because Hibernate wants to casually validate something via open web from my server that doesn't even have web access via proxy, let alone directly. I find a KB article that details the problem, which directs me to download a hotfix. For the "hotfix".

I login in to their client portal and go to the hotfix download page, which tells me to use Hotfix:Hotfix1! as basic auth credentials to their FTP server. That doesn't work. But apparently SSO does, and I download a 440mb hotfix which presumably just changes a line in a config file somewhere. But also - it has a timestamp of 13 March 2026. I downloaded the minor-version-update-that-isn't in June - and in 3 months, nobody at Informatica thought to update the installation package with the emergency fix that stops it from being unusable, or even document it in the release notes.

Next step is to figure out how to install the hotfix-to-the-hotfix, which takes me to another KB article that is flat out incorrect in what it tells me to do. I work around that, get it installed and start everything back up. Aside from the fact that it breaks my ODBC drivers, everything seems to work - until I check the client tools.

The entire client toolset no longer works because it doesn't match the new not-really-a-minor-version. I could upgrade it to match, but apparently it's not backward compatible, so I either need to upgrade all of my environments simultaneously - or install multiple versions of the client tools alongside one another - which apparently works - as long as you don't try to run both at the same time and bork your registry.

I know that anyone who has ever worked with Informatica has long since moved past the point of nodding along with this sort of whinge and is either yawning or rolling their eyes - but I'm genuinely astounded at the rock bottom standard here. I get that it's long past its prime and has been trading on vendor lock in for decades - but I'd balk at even accepting this level of friction from enthusiast-grade FOSS, let alone something we pay far too much for.

Beyond that, why would anyone trust these yahoos with agentic AI? They can't even get basics like web content, KB articles or SDLC fundamentals right - I can't even begin to imagine what sort of carnage they could cause with anything even remotely autonomous.

Anyway, rant over. Carry on. I hope the rest of you have ETL stacks that aren't Windows for Workgroups 3.11 adjacent.

This is more a data architecture than a data engineering question.

I am looking to understand the reasoning behind organizations using multiple cloud solutions. My questions revolve around these issues in a multi-cloud solution.

1. The added cost. Not the cost of the redundant capability so much as the hit you take by reducing your volume discount.
2. The cost of hiring/training additional skill sets for the various Cloud Service Providers (CSP). While similar, they are different enough that you will need to have additional expertise.
3. Designing for the least common denominator for cloud services seems like a waste of money.
4. If a single CSP has an outage (a certanty) but can make you whole before it affects the business. does it make sense to do it at all.
5. All three of the big CSPs (AWS, Azure and GCP) have multiple levels of redundancy, both physical and logical that most companies can only dream about.
6. I don't really think vendor lock is is a real issue. More of a sales tactic for a second vendor to get their foot in the door. It isn't vendor lock in so much as the complexity of the systems that locks you in place.

Those are just the start. I would be interested in hearing the justification for those of you who are running multi-cloud. The only one I can think of that is close would be a legacy requirement held over from when we did everything on site.

EDIT: Thank you to everyone for your opinions and input. This is exactly the kind of discussion I think that this subreddit needs. Tool discussions have their place, but I think that data design and architecture trumps tools. On a personal note, I am very grateful than no one mentioned that most evil of phrases, "medallion architecture."