While Prince Harry grew up going to Balmoral, it’s believed that Meghan has never visited the royals’ Scottish estate

Prince Harry and Meghan Markle were snubbed of another invite to Balmoral Castle in Scotland.

For years, the Duke and Duchess of Sus𝓈ℯ𝓍 have not been invited to the royal family’s summer gathering in the Scottish Highlands, sources tell PEOPLE in this week’s exclusive cover story. King Charles is continuing Queen Elizabeth’s tradition of heading to the Balmoral estate during the late summer, but the monarch hasn’t included Harry and his family this year.

While Prince Harry grew up going to Balmoral, it’s believed that Meghan has never visited the royal family’s Scottish castle.

The 50,000-acre estate, where Queen Elizabeth died in September 2022, serves as a yearly retreat for the royal family out of the public eye, where days are filled with outdoor activities like hunting, fishing and horseback riding.

“It was a life of picnics and paddling around,” Margaret Rhodes, the Queen’s cousin, once said.

The Duke and Duchess of Sus𝓈ℯ𝓍 established a new life in Montecito, California after stepping back from their royal roles in 2020, but they have not been back to the U.K. with their children since Queen Elizabeth’s Platinum Jubilee in 2022 amid an ongoing rift. The strife came to light when Prince Harry, 39, and Meghan, 42, stepped back and has been marked by blistering attacks and open wounds.

For Prince Harry, no single issue is a greater sticking point than that of security, multiple well-placed insiders in his circle tell PEOPLE.

Despite a positive meeting between the father and son in February following the King’s cancer diagnosis, their communication has since deteriorated. Those close to Prince Harry say the sovereign no longer takes his son’s calls or answers his letters.

“He gets ‘unavailable right now,’ ” says a friend of Prince Harry’s. “His calls go unanswered. He has tried to reach out about the King’s health, but those calls go unanswered too.”

Amid the estrangement, the Duke of Sus𝓈ℯ𝓍 is deeply concerned for the safety of his wife and young children, 5-year-old Prince Archie and 3-year-old Princess Lilibet, and he has, according to sources, repeatedly asked his father for help.

“Harry is frightened and feels the only person who can do anything about it is his father,” a royal insider says. Adds another source close to the situation, “Harry is determined to protect his own family at all costs.”

The Duke of Sus𝓈ℯ𝓍 was stripped of his Metropolitan Police bodyguards in 2020 and has been fighting to reinstate police protection for himself and his family ever since. He lost his bid to restore his taxpayer-funded security earlier this year but has plans to appeal.

Insiders in Harry’s circle say that he believes his father has the power to reinstate his security. Buckingham Palace will not comment on security provisions, but a palace source tells PEOPLE the notion that Harry’s security is in King Charles’ hands is “wholly incorrect.”

The issue has created an impenetrable wall between Prince Harry and King Charles, with the conversation shifting from frustration to “complete silence” from the sovereign, claims the friend. For Harry, the echoes of the 1997 death of his mother, Princess Diana, in a high-speed car chase weigh heavily, particularly now that he is a parent.

With both sides firmly entrenched — and Prince William seemingly aligned with the King — any hope for reconciliation appears distant at best. The rift with William is “very bad,” says a royal insider, but it’s not “irreparable.”

Kate Middleton, Prince William, Prince Harry and Meghan Markle at Windsor Castle on Sept. 10, 2022

Some close to the palace suggest that the Duke of Sus𝓈ℯ𝓍’s memoir Spare and public interviews have compromised the family’s trust in him. Others close to Harry counter that if he had proper security, he wouldn’t need to speak publicly in order to help pay for it.

For her part, Meghan “supports Harry 100%, but she wishes he could let go of these lawsuits, be happy and live in the moment,” says a former employee of the couple’s Archwell Foundation.

“She wants him to be free of all of this, but she also knows that because of everything he’s been through and his love for [her and their children], he can’t,” the former staffer said. “She wants him to live in a world where he is not burdened by this.”