Asa Behavioral Change Coach, I hear this question frequently and I have training programs for this very issue of disruptive behavior and interruptions in the workplace, especially in open-office environments.
When your coworker and buddy enters the room, tell him you are working on a deadline and that you would like to spend some good, separate time with him - suggest at break time (ten minutes, just to get the subject matter), or ask him to send you a three line email, of what the urgency may be...then suggest that you can meet later, after work for a drink, coffee, dinner, or maybe breakfast the next morning - whatever your time will permit. Do NOT apologize. He will start to understand that you are not available upon a whim, or his boredom.
It is a good idea to get up, escort him to the door, while you are saying this...but, this can interfere with your flow. Also, consider positioning a plant on your desk, right at eye level, so one will not be so tempted to disturb you, or you do not make eye contact with them to encourage them entering your space.. Never apologize for working, just say you can see him later.
Be warm and sincere, without the dude-get-out-of-my-office-now-attitude...tell him you want to hear him out, but now is not the time.You can be firm, without being harsh. Work on it. Slow down when you speak to him, look him in the eyes...
Call me if you need more info - I have a lot! And good luck.
Bren, 412.999.2149, www.BehaveSense.com
Bren@BehaveSense.com