For when Gentiles who do not have the Law do instinctively the things of the Law, these, not having the Law, are a law to themselves, in that they show the work of the Law written in their hearts, their conscience bearing witness and their thoughts alternately accusing or else defending them, Romans 2:14-15
It's called a conscience, and we're all born with one, whether we're raised in any particular religion or not. We know when we've messed up. How do we know? We feel guilty or ashamed.
Somehow, guilt has gotten a bad rap these days. Don't believe the hype. Guilt is good; a gift of God, even. Not long ago, I heard a popular television preacher telling the thousands of people in his church that, "God doesn't want us to feel guilty." While it's true that a Christian need not be plagued by feelings of guilt over things for which he has already asked and received God's forgiveness, initially, when we do wrong, we most certainly should feel guilty.
God has lovingly designed us with a sense of guilt and shame in order to draw us to Him. Guilt is to our relationship with Christ what a toothache is to our relationship with the dentist. The toothache tells us something is wrong with a tooth, it needs to be fixed, and we'd better get to the person that is qualified to fix it right now. If no one ever felt guilty, no one would ever see his need for salvation and turn to Christ in repentance, without which, salvation does not take place.
So, how do we make the connection between our conscience and the God who created it? How does our having a conscience prove that God exists? Well, it does require some introspection, but for anyone who will take a few minutes to sit down and think about why he feels guilty over his wrongdoing, the answer will become apparent.
So what other explanation could there be for the guilt we feel over our wrongdoing which is not breaking the law, not hurting anyone, and which no one else will ever know about?
because that which is known about God is evident within them; for God made it evident to them. Romans 1:19
Our hearts know, even if we don't want it to be true, that there is a God.