I totally agree how he went about it was wrong and if it was a shared permission then that's a little different; I could just imagine how I'd see it if it was my permission and people I'd brought with me started yoinking neighbouring permissions without cutting me in... But if it's as you say sounds like you should simply just stop ferreting with him. If he needs you more than he needs you it'll impact him more.
It's supposed to be a pleasant activity remember?!