How to smooth the edges of a black and white mask?

Question

Graphic arts

As you can see, the edges of the mask are rough and, accordingly, the image cut out by the mask is also obtained with sharp edges. I know the problem seems simple, but I spent a whole day and couldn't solve it because of my inexperience.

Answer 1

A 3px Gaussian blur of this toothy image in its entirety should help.