I've tried what you said about the substeps. Here was the result that I got.
I noticed 1 thing is that everytime the punch start lifting up after reaching the targeted displacement, the model begin to fail and the solver pivot warning keep showing up.