I've been trying to make a coastal border for this:

However when I use offsets or thickness of the line I get this weird result:

Any other ways to do this in a way that I can specify the distance from the coast?

There´s not much you can do about it as this is how it works. You can test it with a duplicate of the map and give it a fat stroke. Then go Path->Stroke to Path - then Path->Break apart and then Path->Union - kill fill and add stroke color:

There's a live path effect for this. Select your shape then [shift+ctrl+7] to open the LPE dialog. Click [+] to add an effect. Choose [Offset] and set the offset distance. Positive values draw an outset shape and negative values draw an inset.  The default zero value reproduces the original shape. There are several join types available. Use [Join: Miter] to give an angular shape or use [Join: Round] to produce something like your initial effort.

I tried this Offset LPE but it produces pretty much debris (cyan stroke:) which can be difficult to clean up later. Much more reliable is Path->Outset a few times imho.

Removing internal artefacts is straightforward: 
    [Path > Object to Path] or [ctrl+shift+c] 
    Node tool [n]
    Select one node on the outside path 
    [Path > Reverse]
    [Path > Union] or [ctrl+shift+=]