Meet confidence. And its boisterous cousin overconfidence.

A leader who “ums” and “ahs” over too many decisions probably won’t be a leader for very long. Even when uncertainty is warranted, an obvious lack of confidence won’t inspire your team or coworkers. Professionals know that confidence can have tremendous value, but where exactly does it come from? And more to the point: how can we get more of it? After all, when we teach children that “you can be anything you want”, we do so on the premise that confidence is a key driver for success in life and work.

Research shows that most of us are not just confident, but overconfident. One influential study from 1977 that measured knowing and certainty asked participants to answer a series of questions then record their confidence in the accuracy of their answers. What the researchers found was that people who were 70% confident were right 60% of the time; while those who were 90% confident were right just 75% of the time. Confidence only gets you so close to the truth.

A person’s lack of confidence, meanwhile, is generally seen as a problem that needs to be fixed. There are books, courses, and coaches whose goal it is to help people become more confident. For good reason: without confidence, we will never believe in ourselves or have the nerve to try things that test our abilities.

The tricky part is getting your confidence and ability to properly align.

How do we align confidence and ability?

Confidence is most useful when it emerges from a foundation of ability. Research has given us some guidelines for how to make that happen. For starters, we have to be experienced in the domain we’re operating in. Reliable, grounded confidence requires that you already know the ins and outs of the system or specialization you’re attempting to predict or control or explain.

Second, that domain or system must be regular and predictable. You can certainly be confident in your ability to perform heart surgery, but it’s meaningless to say you’re confident you can win the lottery. Twenty years of experience buying lotto tickets doesn’t translate into accurate predictions for your next bet. After all, studies of investors and investing have found that even experienced stock traders often perform at a level just slightly above chance.

One implication of all this is that experts in stable fields can learn to trust their own intuition. A doctor making a snap diagnosis, a pilot navigating rough weather, or a designer gauging next season’s trends are rightly justified in trusting their instincts. But be wary of the experienced sports gambler who claims some underdog will win, or an investor who promises high returns year after year.

When predictions fall flat

Nobel Prize-winning psychologist Daniel Kahneman wrote about the hazards of confidence during his years in the Army. Tasked with identifying troopers with leadership potential, Kahneman put groups of soldiers through an obstacle course where he observed who took charge, who came up with solutions, and who became discouraged when their opinions were rejected by the other soldiers.

By the end, Kahneman believed he had a good idea of who had the qualities of a leader. But when he measured the results of his predictions? Barely above chance. “The story was always the same: our ability to predict performance at the school was negligible. Our forecasts were better than blind guesses, but not by much.”

Despite the discouraging feedback, Kahneman noted that he continued to feel confident in the predictions. “The statistical evidence of our failure should have shaken our confidence in our judgments of particular candidates, but it did not. It should also have caused us to moderate our predictions, but it did not. … I was reminded of visual illusions, which remain compelling even when you know that what you see is false.”

Gut instinct can be powerfully persuasive. Here’s a question: a bat and ball together cost $1.10. The bat costs $1 more than the ball. How much does the ball cost? For most people, $0.10 comes screeching to mind. That intuition is fast, easy, and just feelsright. Except that the correct answer is $0.05.

Confidence is a feeling that needn’t require logical analysis. It can accompany a gut reaction or even just a sense of ease. Take, for instance, the evidence that people are more likely to invest in stocks with easy-to-pronounce names. In Kahneman’s words, “Confidence is a feeling, which reflects the coherence of the information and the cognitive ease of processing it. It is wise to take admissions of uncertainty seriously, but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”

Sorting fact from fiction

What about overconfidence, confidence’s boisterous cousin? It certainly has its advantages: it motivates us, it can be self-fulfilling (believing in yourself makes you more effective), and it can certainly convince others to trust or follow you.

But that’s not to say that overconfidence should be taken to the extreme. Rather, that it’s useful in motivating ourselves and others. Surely we can be ambitious while also aware of our limitations. As we try things, and experience different rates of success and failure, we should learn and adjust our confidence level accordingly. If we don’t believe that our flaws exist, it won’t be easy to fix them.

The bottom line for professionals in leadership roles is this: seek balance. If you come across as overly indecisive, you’ll of course fail to inspire confidence in your colleagues or friends. On the flip side, brash overconfidence coupled with errors and mistakes does its own brand of harm. Instead, aim for an accurate understanding of your abilities coupled with a healthy skepticism for gut instinct. And don’t forget that doubt is a necessary element in effective, inspirational confidence.