Remove listeners on directive $destroy

[nhim175]

For now, even when the directive being destroyed, setPosition is still getting called.