A few days ago, I stepped into a trap ! This made me lose time for fixing things (and even more for writing this post...). In the hope that you will avoid my mistake, I am sharing this war story. I also obviously opened a bug, more about this below.
TL&DR: be careful when using CHANGE MASTER TO MASTER_DELAY = N: it might wipe your relay logs !
As written in the TL&DR, running CHANGE MASTER