It isnt your fault, youre doing nothing wrong.. in fact youre going out of your way. Each time he was drunk or at least drinking and that will cause it to be more difficult to “complete”.. He shouldnt be blaming you.
Tell him not to drink before hand..