I will frame the question in terms of a specific C# objective that I am working on right now but I imagine the question is a pretty general one related to the Dunning-Kruger effect, in a way - how do you know how to build an application when you don’t know all the issues you are supposed to prevent?
There is a message hub and I am developing a consumer for it. The original plan was to just create a few background services that get initialized alongside the application, that have a loop to load new messages/events and then process them.
Some time has passed and it feels like I am knees deep in Wolverine, Quartz, Hangfire, MassTransit, transactional outbox and all manner of different related things now. The potential issues are dealing with downtime, preventing loss of messages by storing them in a separate table before processing them, and everyone on the planet has a different idea on how to prevent and solve them and none of them sound simple and straightforward.
Honestly at this point I don’t know enough about which problems are going to appear down the line and if I need to use third party libraries, but I am guessing they exist for a reason and people aren’t supposed to just manually create their own amateurish implementations of them instead? But how do you know where to draw a line when you don’t know exactly the problems that you are supposed to be solving?
What are the problems with having a table for the message queue over a whole 3rd party library for it, or what’s wrong with the MS BackgroundService class? How are you supposed to know this?
Aside from the commonly stated - experience:
The key point to keep in mind is that, at the end of the day, your building an application to satisfy a customer. Or to be exact, a list of requirements that may or may not be constantly changing. In this case from what i’m reading, I am assuming its your hobby project, so the ‘customer’ is you.
In this case, over-engineering is when you add more functionalities/services than what you needed to hit the bare minimum requirement. Ideally you want to hit basic requirements first, then start designing/engineering on top of what you have when your customer wants more features, etc.
Your design providing more features than the bare minimum should be an ‘accident’ more than intentional ideally, unless you think the extra feature takes 0% effort to implement. (Though TBH safe rule of thumb is, never design for additional unrequested features)
With the above context in mind, you should be asking the questions:
Yea, welcome to client engineering. Usually its handled by senior developers or project managers, unless your in startup.
TLDR; It’s not about finding a design that is perfect, its about finding a design that is acceptable by the involved. What is acceptable? that is for the involved parties to decide.