when brushing colour back in black-white the colour disappears as soons as I let go of the cursor

I just started the trial version of PS. I made an adjustment layer in black and white. When I brush to get some of the colour back, the colour disappears as soon as I stop brushing. What do I do wrong? Thanks!