If ”letting go”, which is mere forgetfulness, would bring about the end of mental afflictions, ordinary sentient beings would have all achieved nirvana a long time ago, because ordinary sentient beings are all forgetful of their actions, of their mental afflictions, of their vows and commitments, and so forth.
Now, a contributory cause or factor which does enhance mental afflictions is inappropriate attention, and not paying attention to an object which increases afflictions does help reducing their power temporarily, until they are fully exterminated by an actual antidote. This is the advice of Shantideva in his Guide.
However, mental afflictions, such as ignorance, desirous attachment, anger, and so forth, do not go away by just ”letting go”. They have to be actively eradicated by their temporary antidotes, such as love, compassion, and bodhichitta, and by their ultimate antidote, direct, non-conceptual realization of emptiness.
Assuming that mental afflictions cease by themselves by just ”letting go” or forgetfulness is the evil advice of the Chinese Ch'an monk Hashang Mahayana, soundly defeated by the Indian master Kamalashila in the famous debate of Samye, under the eyes of the great king Trisong Detsen, the emanation of Manjushri.