Pay down mortgage?

Hello all and happy labor day!

Here's my dilemma ( and it's a good problem to have):

I have almost 80% of the cash needed to pay off my mortgage of 5.12% over 30 years. I'm trying to decide whether to a.) use all of it to pay against the mortgage, or b.) use half of it and invest the other half in PMs.

I know some of you might say " invest all of it in PMs " but I would feel really foolish if I lost this money that I have worked so hard to save of. My job industry is very fickle and the idea of having my home paid off is really attractive. Also, are there any legal or other reasons not to pay it off or down?

Thanks