I've started meddling with the distort-> roughen effect on images and I've noticed that every time I do so, there seems to be an automatic square boundary made. It makes my image distorted but confined to a square boundary. How do I make it not look like its in a box?

I´m not able to recreate what you´re describing even when I add the Roughen effect to an LPE stack.

Can you upload an example file that shows that issue?