This sounds exactly like what my parents/my ex church members would have said. i.e. You can go ahead and do that just not anywhere around us. We can't afford to be associated with that sort.
Being told you're delusional does not necessarily mean you're mental.
