In a photograph taken moments before the attack, the alleged gunman can be seen standing behind Mr Abe in a grey t-shirt with a black bag
The alleged gunman, named as Tetsuya Yamagami, 41, believed Abe was part of the group and shot him for that reason, they said, without naming the group.
Abe died in hospital on Friday morning after being shot while speaking at a political campaign event.
Yamagami has admitted shooting him with a homemade gun, police said.
Abe was Japan's longest-serving prime minister and his death at the age of 67 has profoundly shocked a country where gun crime is very rare.
He was shot while campaigning for his former party, the Liberal Democratic Party (LDP), in the run-up to upper house parliamentary elections on Sunday.
Prime Minister Fumio Kishida, also an LDP member, said he was "simply speechless" at the news of Abe's death, vowing that Japan's democracy would "never yield to violence".
He said the election campaign would continue on Saturday with tightened security, with Sunday's vote still set to go ahead.
Police are still investigating why Abe was specifically targeted and whether his killer acted alone.
He was giving a speech on behalf of a political candidate at a road junction in the southern city of Nara when the attacker struck him from behind.
Photos from the event show the suspect standing close to Abe moments before he was shot.
Witnesses described seeing a man carrying a large gun moving within a few metres of Abe and firing twice. The former prime minister fell to the ground as bystanders screamed in shock and disbelief.
Security officers dived on to the gunman who made no attempt to run.
The weapon that killed Abe was made using metal and wood, officers said, and appeared to be wrapped in duct tape.
Several other handmade guns and explosives were later found at the suspect's home.
Abe suffered two bullet wounds to his neck and damage to his heart during the attack.
He was said to be conscious and responsive in the minutes after the attack, but doctors said no vital signs were detected by the time he was transferred for treatment.
Medics worked for hours to save him before he was pronounced dead at 17:03 local time (08:03 GMT) on Friday.
At around 06:00 local time on Saturday, a hearse carrying Abe's body was seen leaving the hospital to return to his home in Tokyo.
The improvised weapon lies on the ground after the alleged gunman was arrested
On Japanese social media, the hashtag "We want democracy, not violence" was trending throughout Friday, with many users expressing their horror and disgust over the incident.
Gun violence is extremely rare in Japan, where handguns are banned and incidents of political violence are almost unheard of.
In 2014, there were just six incidents of gun deaths in Japan, compared to 33,599 in the US. People have to undergo a strict exam and mental health tests in order to buy a gun. Even then, only shotguns and air rifles are allowed.
Abe first held office for a year in 2006 and then again from 2012 to 2020 before stepping down, citing health reasons.
While he was in office, he pushed more assertive policies on defence and foreign policy and had long sought to amend Japan's pacifist post-war constitution.
He also pushed for an economic policy that came to be known as "Abenomics", built on monetary easing, fiscal stimulus and structural reforms.
He was succeeded by his close party ally Yoshihide Suga, who was later replaced by Mr Kishida.
The suspect was confronted by a police officer following the shooting
Prominent voices across the world condemned the incident, with UK Prime Minister Boris Johnson calling it a "despicable attack".
US President Joe Biden called Prime Minister Kishida to express his "outrage, sadness and deep condolences" on Abe's shooting. He said the assassination was "a tragedy for Japan and for all who knew him [Abe]".
Mr Biden ordered US flags to be flown at half-mast over the weekend.
In Brazil, President Jair Bolsonaro declared three days of national mourning, expressing outrage and sadness, while Indian Prime Minister Narendra Modi described Abe as a global statesman.
South Korean President Yoon Suk-yeol called the attack an "unacceptable act of crime" and extended condolences to "the Japanese people for having lost their longest-serving prime minister and a respected politician".
Chinese Foreign Ministry spokesman Zhao Lijian said his country was shocked by the attack.
He added that "this unexpected incident should not be associated with Sino-Japanese relations" and had no comment when asked about Chinese social media reaction.
Comments gloating over the attack on Mr Abe have dominated Chinese social media and have also surfaced on Korean platforms.
China and South Korea have historically had complicated and fraught relationships with Japan. Abe, known for his military hawkishness, was unpopular with citizens of both countries during his term in office.
-- Courtesy of BBC News