From what I understand, we can only collect merit from the merit field (such as from temple, carry out assignments by qualified Lamas, etc), and with correct motivation, i.e. towards Buddhahood, which is ALWAYS for the benefit of others. If we have motivation to become Buddha, which is always for the benefit of others, how can that be selfish?
So if we build temples but our motivation is to have more wealth for our family enjoyment, technically, we do not collect any merits, though we would collect good karma.
So, technically, you cannot collect merits and be selfish at the same time