The broker may not have a say so in this. Once an offer has been accepted the deposit belongs to the seller. Does your contract have a contingency time frame to accomplish loan approval? Are you within that time frame? What is the reason the deposit is not being returned? When a bank has denied you a loan, find out why and try getting approved with 1 or 2 more banks. Loans are tough but not impossible. Different brokerages and banks have different investors.
If you believe your position is correct and legal try and negotiatiate for the return. If all else fails have an attorney write a letter to the broker and the seller. Court action or arbitration may be necessary. Good luck.
PS I am seeing a lot of this situation in Souhern California. Especially with bank owned properties. They have found another way to recoup losses.