The geometry has two sharp interior corners. Each one creates a stress singularity in the model, which means a point of theoretically infinite stress. Since the elements are of finite size, a finite stress value is computed. As the elements get smaller the stress gets larger, without limit. The only reason the stress is higher in the model with contact is the elements are smaller.
The corrective action is to add a small blend radius to the two sharp corners and use elements small enough to go around the blend. That will eliminate the stress singularity from the model. It is best to avoid using contact elements in the area of high stress gradients.