When is it time to consider a system redesign?

Originally posted in r/developer but I thought this was a more active community.

So currently in my company, I inherited a project to manage. I am fairly new to the industry (about 4 years at this point) and I’ve never lead a project. However, I do find that the underlying engine of this project has a lot of breaking edge cases that our product owner keep finding.

To make matters worse, the engine is barely testable. Our only way of testing the engine is taking whatever input “model” that comes from the user and shoving it through the engine and test certain states afterwards.

We are currently doing consulting work using this system, and of course, the system keeps breaking, and we keep patching (as best as we can), all with constant deadlines.

Personally, I think the underlying algorithm in this engine is a bit misguided. I also think that the engine can be broken into testable subsystems. Any advice?