"Cast" to TV only working sometimes?

I have a Samsung MU6100 TV and a Lenovo P2 phone.  When I try to screen mirror via wi-fi direct / miracast (using the Cast option on my phone and choosing the Samsung TV from the list), most of the time it doesnt work. I just get a still image of my phone screen appearing on the TV. Any actions made on the phone are not mirrored. But, sometimes, once out of every 5 times or so, it works fine, perfectly mirroring my phone actions on the screen.

Has anybody else had this issue? Is there something I can do to make it work consistently, instead of intermittently? Is is it a problem with the TV or the phone?