I have not removed it yet, but I know that the front part of the handbrake slot is a separate piece which I have pulled up from around the handbrake hole. If you can pull that off it might give you enough space to get the phone.
I have a 2008 mazda demio (manual transmission) and while trying to change the head unit I found a pin adapter/connector by passenger side near to the gear stick but by the foot area. Anyone can tell me what that's for ?
carefully pull straight up, it is held in place with Christmas tree clips, but be careful not to pull too hard as you will need to disconnect the harness plugs at the aux jack and 12V power outlet