Unfortunately, this is a "feature," but try filtering down your unnecessary phonebook entries if possible, or do what Skorpio suggested, try a full repairing, and see if it still happens.
My Nokia 1020 Win 8.1 phone takes about 30 secs, but my wife's same phone take about a minute, she has way...