Some of the reasons for this are glaringly obvious, while others exert a more subtle force. John Grindrod is the author of Concretopia: A Journey Around the Rebuilding of Post-War Britain. For hundreds of thousands of years, our confidence has been shaped by the environments that we are allowed into or not allowed into, he says, pointing out that, by its nature, castle design led to the feeling that those inside were protected by its architecture, while those outside were not. After the war, Grindrod notes, this began to shift. We saw a desire to try to create buildings that were more transparent and more permeable, he says. An egalitarian architecture as a panacea to a lot of issues around people feeling very disconnected from power.

But the issue is that we do not live in an egalitarian society. The design of a public school such as Eton has much in common with, say, the colleges of Oxbridge, as well as the Inns of Court and the Houses of Parliament. If you grow up among these kinds of buildings, you are not only less likely to be daunted by their grandeur but ,on the contrary, you will feel at home, as if you belong there and they speak your language. When the competition to build the Houses of Parliament came along in the 1830s, you were only allowed to enter buildings that were neo-gothic or Tudor, adds Grindrod. People who understood this vernacular, of course, would have been to Oxford and Cambridge and all those other hallowed institutions.

There have been architectural ripostes to the established elite, however. Maconie speaks fondly of St Georges Hall in Liverpool, a neoclassical building begun in 1841, when the city was flourishing: Its designed to be the first thing you see when you get off the train at Lime Street, this grand edifice, and its supposed to say, Were not bowing to anyone, were supremely self-confident and were as good a city as anywhere in the world. You see that in a lot of Manchesters cottonopolis-era architecture. A sort of swagger in bricks.

Swagger is one of those words often used to describe confident northerners particularly men. I think of the self-confidence of the north in terms of, say, the Gallagher brothers [from Oasis], that and Arthur Seaton [from Alan Sillitoes Saturday Night and Sunday Morning], says Maconie. That kind of self-confidence is born, to a degree, of failure. You get a lot of street confidence in northern males, its an Im never really going to make anything of myself in terms of money or power or prestige, but I can enjoy the prestige of being the loudest guy in the pub.

Confident women, meanwhile, often find they are described as bossy or snobby. Katty Kay presents BBC World News America you may remember her as the presenter whom Dr Ben Carson, the former candidate for the Republican presidential nomination, tried to silence during a live TV discussion of Trumps alleged sexual assaults, asking for her microphone to be turned off. She is also the co-author of The Confidence Code: The Science and Art of Self-Assurance What Women Should Know.

Kay is the daughter of a diplomat, she attended Oxford, and worked for the Bank of England before beginning her career in journalism. Despite this grounding, professional confidence has been a quality that has often alluded her, and she attributes its cause to thinking I wasnt bright enough, and I was conscious of not being confident enough.

But she is aware that she is not alone. Evidence of women underestimating their abilities is comprehensive and across the board, she says. It exists in sports, it exists in politics, it exists in business, it exists in the military. It is quite the reverse for men. One of the most reliable social studies you can do is to give men and women a scientific reasoning quiz, she says. Men tend to overestimate their abilities by more than 30%. Women routinely underestimate their abilities. In reality, the quiz results reveal men and women tend to do about the same.

This, of course, has implications for both an individuals career and the workplace in general. Hewlett Packard has done work on promotions, Kay continues. Women will apply for promotions when they have 100% of the skill set, men will go for those same promotions with 60% of the skill set, because they figure theyre going to learn the rest when they get there and theyre right, they will, and so could we. Its one of the biggest factors I think in why women hold themselves back at work. Now, there are lots of structural reasons, the playing field is not level, but we are also not going for those promotions, were not asking for those pay rises in the way that men do.

During the last few months I have been making a radio series about confidence what it is, where it comes from, why some of us have it and others dont, and what to do about it if your confidence levels are in short supply. I should note that I am not a confident person. I spent my entire first term at primary school allowing myself to be called Louise because I was too shy to tell them my name was actually Laura. I also recently gave a talk at a festival and, for fear that I was taking up everyones valuable time, began early, then garbled through it at high speed and low volume, apologising frequently. I did not ask for a lectern, or for the window on to the noisy street to be closed, I did not allow myself to stop and breathe, because I feared that to do any of these things things that would have benefited both the audience and myself might have been considered arrogant.

It seems to me that confidence has much to do with space with how much room you feel able and allowed to take up. Grow up in a detached house with several acres and you might feel entitled to more room than someone raised in a terrace or a high rise with a tiny balcony. Attend a school where the class sizes are smaller, where fees are paid, and the buildings are grander, and you will learn early that you have a right to spread out, raise your voice, ask for more.

To muddy things further, girls are raised to believe that being smaller is preferable; in a hundred thousand ways we receive the message that we should be quieter, thinner, less demanding, in case we are deemed bossy, or our views too strident, or in case a man asks for our microphone to be turned off. To ask for a pay rise, then, is demanding; it says I am worthy of more and to women, who have spent their lives being told that they should be less, this is conflicting. Men, meanwhile, are raised to be go-getters, to conquer and to win.

But, male or female, we are all a mess of contradictions: the business leader who cant make small talk, the party animal who balks at intimacy. I feel relatively self-assured so long as you cant see me so I can write an article, or present a radio programme, or be as cocky as you like on email, but in the decade that I worked in the Guardians offices, it filled me with dread to have to walk over to speak to my editor.

In the making of this series, there have been moments when I have begun to question whether confidence is such a marvellous thing at all. I dont know if I always trust it, and certainly I have wondered whether confidence always has to equate with brashness whether there might not be a quieter, gentler form of self-worth. I have thought often of something Maria Konnikova, author of a book about con artists, The Confidence Game, said to me: I have to be very wary of people who speak confidently. That is actually a sign that you should be a little bit more sceptical of them. And Ive considered the state of the world and wondered whether maybe all the big mouths and hot-talkers should just pipe down for a moment. I certainly look around me at the world and see strong, confident men who seem to be leading us into very dark places, Maconie notes. Isnt quiet, modest competence a better thing? Ease in ones own skin, I think, is a different matter. To not feel beholden to anyone or inferior to anyone, thats hard-acquired, I think, and that comes from a long immersion in what you do. Sometimes a little more discretion and humility might be a good thing.

Susan Cain is the author of Quiet: The Power of Introverts in a World That Cant Stop Talking. She cites a recent study by the Kellogg School in the US which found that in an average large meeting, three people do 70% of the talking. And thats horrifying, she says, because if you imagine it, everyone in those large meetings is equally likely to have good ideas but were only hearing from three of those people. That is just so much power and mind talent that has never seen the light of day.

The problem, she says, is that we have created a culture in our schools and workplaces where those people who are just more vocal, who are more dominant, more willing to take up space are automatically accorded all kinds of advantages, both consciously and unconsciously. But if you consider that a third to a half of the population is introverted, perhaps it is time for us to change the culture rather than change ourselves.

Still, we have grown accustomed to trying to change ourselves. Visit the self-help section of any bookshop and you will find any number of guides to gaining confidence: Susan Jeffers Feel the Fear and Do It Anyway, Paul McKennas Instant Confidence, Russ Harriss The Confidence Gap among them. One of the bestsellers is Bren Browns Daring Greatly. In 2010, Brown, a research professor at the University of Houston, gave a TED talk called The Power of Vulnerability which has gone on to be one of the most-viewed TED talks of all time (31,649,423 views at time of writing). Browns theory is that we acquire true confidence through vulnerability. Courage starts with showing up and letting ourselves be seen, she writes.

The School of Life, the educational company founded by Alain de Botton, takes a similar approach. It runs a popular confidence workshop and publishes a guide, On Confidence, that draws on the wisdom of Erasmuss 1509 essay In Praise of Folly, and suggests that a willingness not only to be vulnerable but also to be a fool is crucial to evolving greater self-worth. Theres a type of underconfidence that arises specifically when we grow too attached to our own dignity and become anxious around any situation that might seem to threaten it, it states. We hold back from challenges in which there is any risk of ending up looking ridiculous, which comprises, of course, almost all the most interesting situations. The happy news is that, far from regarding it as an elusive gift, confidence is rather a skill based on ideas about our place in the world, and its secrets can be learned.

Katty Kay, agrees. I see confidence almost like building blocks, she says. Its almost a tangible physical commodity. You get confidence by doing things and trying stuff thats hard for you and when you do those things its like you bank a bit of confidence, you put it in your confidence wall. Not so long ago she was called to a meeting on Middle East affairs at the White House. And I thought: Oh my God, Im a fraud, all these people are super-duper experts, what am I doing here? Im just a generalist! When they reached the Q&A part of the meeting, Kay noted how the men in the room just jump in with questions, and Im sitting there thinking to myself: I must ask a question, I cant be one of only two women and neither of us ask questions! And eventually I think: For Gods sakes, Katty, youre nearly 50! Put your hand in the air and ask a question! So I put my hand up, and the question comes out, and the Earth didnt open up and swallow me whole. And the next time I was in that situation it was that bit easier because I had banked a bit of confidence.

Its an approach echoed by Brown. Courage is a habitus, a habit, a virtue: You get it by courageous acts, she writes. Its like you learn to swim by swimming. You learn courage by couraging. The brain, after all, is not rigidly set, but malleable and open to change, and so we can learn to be bolder through repetition and reward.

A 2014 study at Dartmouth College, looked at the role of the frontostriatal pathway, which connects the medial prefrontal cortex, implicated in self-knowledge, to the ventral striatum, which provides feelings of reward and motivation. Researchers used magnetic imaging to measure both the physical parameters of that pathway, which it termed the road and the activity levels on that pathway, termed the traffic.

Participants answered questions about how they rated themselves in the short and the long-term with regard to qualities such as happy, hard-working, pessimistic and depressed. The researchers found that an individual with a strong road was likely to experience higher long-term self-esteem. Higher traffic levels on the pathway, meanwhile, showed momentary rises in self-esteem. They also only saw traffic when participants rated themselves with positive qualities, not negative ones. So if we think about ourselves positively, the areas of the brain connected with motivation, pleasure and reward are stimulated.

Just like mastering any other talent, gaining self-assurance requires repetition and time, writes Dr Stacie Grossman Bloom, a neuroscientist who has examined the role that neuroscience can play in raising confidence. The first step is to push back against the obstacles we know stand in our way by being mindful of the situation, and deciding to be confident. Making that complex decision is a multi-step process that taps into our emotions and engages many other parts of the brain. It doesnt matter what level of self-assurance you start at, the more time and effort you dedicate to practicing being more confident, the faster your brain will change and the faster youll master it.

At the Impact Factory in north London, Jo Ellen Grzyb runs workshops on communication, negotiation and public speaking. Over the course of her career she has developed her own tricks for pushing back against obstacles and mustering confidence. If, for example, you find yourself in a meeting in which only three people are blathering on, you might consider interjecting for the good of your colleagues. You put on your Superwoman or Superman cape and you are rescuing everyone else, she says. Because if Im thinking, I have to speak, I have to speak, what am I going to say?, Im all in my head. But if I think, I can rescue this meeting, then that builds my confidence because Im not just doing it on my behalf, Im doing it for the whole room.

There are physical tools, too. You think you dont have the confidence to interrupt this blusterer, she says. But if you begin to speak and you give eye contact to everybody but that person, its one of those little tiny magic tricks, because that person is being ignored. Its not being rude, but you can change the dynamic very quickly. Speak, make eye contact but not with the person who is taking up all the space.

Among many roles, Patsy Rodenburg is head of voice at the Guildhall School of Music and Drama in London, and works with actors, teachers, world leaders and members of the corporate world, teaching on matters of voice and presence. Although I cant talk about the psychology of confidence, I know what it looks like in the body, and the breath, and the voice and the pace, she says. Often people who are trying to be confident and arent swing the pendulum the other way and theyre too loud. They take up too much space. Others are collapsed in their bodies. They dont want to make eye contact, so there is a withdrawal from the world. You disappear. You stop breathing. Its the equivalent of the mouse with the hawk above it.

Her advice is that there is no overnight fix for the underconfident. It takes consciousness, choice, but also simple exercises that might have to be done for the rest of your life. Technique is for the moments when youre upset, disturbed or fearful. She asks people where they feel uncomfortable in these moments. All these tensions stop us breathing, she says. And breath is the fundamental thing in using our voice and connecting to people. So we have to get the breath low and deep and not rushed.

For a lot of women, its a matter of lifting the sternum, for others it might be finding some kind of external connection. I might be sitting at a desk feeling scared, she suggests. So Im just putting my hand against a desk and Im just gently pushing. And if you push against the desk, and your feet are on the floor, you can re-set the breath. Its about re-setting. Youve just got to come back into yourself.

Conversely, coming back into yourself is often a matter of stepping out of yourself. Somebody who is incredibly confident has authority and stillness and theyre interested in us, Rodenburg says. Real confidence has gravitas. And when were fully present, were interested in something outside ourselves. So one of the best things you could do if youre not feeling confident is just listen to others, and be attentive.

Once, I thought gaining confidence might require me to become someone else entirely someone harder and louder and more bruising. But really I think it is a matter of stepping beyond yourself; an adventure of sorts, into the unknown and the brilliantly possible. It is about taking up as much space as you need. About daring to be wise. And, if necessary, its about keeping a steadying hand on the table.

The Confidence Trick, written and presented by Laura Barton, begins on 30 October at 8pm on BBC Radio 4.