I often see this verse interpreted as David referring to his son as being in heaven. Now I'm not saying his son did not go to heaven, but all David is saying here is that he too will die and be in Sheol.
"But now he is dead. Why should I fast? Can I bring him back again? I shall go to him, but he will not return to me.”
-David, 2 Sam. 12:23(ESV)
"I wish it were in my power, after this day, to cry down all love but the love of Christ, and to cry down all gods but Christ, all saviours but Christ, all well-beloveds but Christ, and all soul-suitors and love-beggars but Christ." -Samuel Rutherford, Letter CCIII. to William Gordon at Kenmure