By Gregor Stuart Hunter 

China's stocks fell Tuesday, casting doubt on the potency of Beijing's aggressive rescue efforts, while investors elsewhere in Asia appear to shrug off concerns over Greece's debt situation.

The Shanghai Composite is down 1.3% at 3727.12, with 864 stocks falling and 55 making gains Tuesday, according to Chinese data provider Wind Info. The Shenzhen Composite fell 5.3% at 1932.83, while the ChiNext board, composed of smaller cap stocks, and sank 5.7% to 2352.01. All indexes are off more than a quarter from June highs.

Shanghai's relatively softer decline follows guidance over the weekend for a rescue fund to invest in exchange-traded funds tracking blue-chip stocks. Gainers, concentrated in the insurance, banking and infrastructure sectors, included China Life Insurance Co., Bank of China Ltd. and China Railway Construction Corp., all of which hit their daily trading limit after shares rose 10%.

Some analysts attribute Tuesday's losses in China to margin calls, when brokerages call in money owed by investors who borrowed to buy shares. "The pressure on margin calls remain heavy," said Qian Qimin, an analyst at Shenyin Wanguo Securities. While some highly leveraged Chinese investors have already been forced to liquidate margin positions in small and midcap stocks, the selloff could continue.

"The entire clearance process, in our view, may last a few more weeks," said Nomura in a research report.

China has rolled out a steady stream of measures to arrest the selling frenzy that knocked $2.4 trillion in value from China's equities over the past three weeks. According to data from FactSet, 514 stocks are suspended in Shenzhen and 162 in Shanghai. Despite the recent rout, Shanghai shares are up 82% over the past year, and 16% since January. Shanghai recovered modestly Monday, which some investors and analysts attributed to heavy buying of blue-chip stocks by state-backed funds.

That buying looks set to continue. Chinese brokerage Haitong Securities Co. on Tuesday said it would boost the amount it invests in blue-chip exchange-traded funds by 15 billion yuan ($2.42 billion). The announcement follows an agreement over the weekend among 21 brokerages to invest in a stock-rescue fund.

Other measures include halting new initial public offerings, raising quotas for foreigners to buy stocks and an initiative by the central bank to provide funds to help investors borrow to buy shares.

Some brokerages are encouraged by Beijing's efforts: "Chinese monetary and fiscal easing policies, combined with further reform measures, should support [mainland] A-share indexes to new highs in 2016," says Raymond So, co-head of research at Chinese broker CCB International Securities. A number of sectors stand to benefit from the Beijing-backed call to buy, including financials, property developers, health care, utilities and telecoms, he added.

Still, China's volatility is starting to spill into global markets, as Beijing's moves to arrest stock selloffs have darkened the outlook for the world's second-largest economy. Oil prices tumbled nearly 8% Monday, their biggest single-day decline in more than three months, amid fresh fears about weaker demand from China, one of the world's largest consumers of raw materials. The sharp falls moderated Tuesday, with Brent crude gaining 0.9% to $57.05.

Stocks in Hong Kong extended a selloff after their biggest fall in three years on Monday, with the benchmark falling 1.2%. A gauge of Chinese companies with Hong Kong listings, known as H-shares, was down 2.9%. The drop from its high in May briefly put the index in bear market territory, defined as a 20% fall.

Elsewhere, Asian markets rose as investors appear hopeful that a resolution between Greece and its creditors is still within reach.

"It seems that investors either believe that 'Grexit' can still be avoided, or that, if it does happen, the contagion will be limited," analysts from Capital Economics wrote in a research report.

Greece's vote on Sunday to reject creditors' demands, including pension cuts and tax increases, could put Greece closer to exiting the eurozone. That led markets in Europe and the U.S. lower, though the declines weren't as dramatic as expected. The euro is unchanged against the U.S. dollar, while the yen gained 0.1% against the dollar.

The Nikkei 225 Stock Average rose 1.3% while South Korea's Kospi Composite fell 0.7%.

Australia's S&P/ASX 200 rose 1.9%, its strongest one-day gain since mid-February, with falling oil expected to lower operating costs for energy-intensive commodity producers. The Reserve Bank of Australia kept benchmark interest rates on hold at 2.00%.

Meanwhile, banks warned that Malaysia's financial markets could come under further stress, after the country's currency fell to its weakest level since the Asian financial crisis Monday. The oil-exporting nation has been hit by falling commodity prices, which have nearly halved since July last year. Malaysia's stocks are down 1%.

The ringgit strengthened to 3.8060 against the dollar amid speculation that Malaysia's central bank may be intervening to buoy the currency, traders say. The currency is down 8.2% for the year, making it one of Asia's worst-performing currencies.

"Among Asian financial assets, Malaysia's are the most exposed to oil," analysts from ING wrote in a research note. Efforts to strengthen the ringgit "will give way if oil prices continue to decline," the bank added.

Malaysia's currency also has come under pressure as political strain builds against the country's prime minister. The Wall Street Journal earlier reported that Malaysian government investigators looking into the activities of state investment fund 1Malaysia Development Bhd., or 1MDB, had traced almost $700 million in deposits into what they believe are Mr. Najib's personal accounts. Mr. Najib has denied wrongdoing.

Anjie Zheng, Yifan Xie and Dominique Fong contributed to this article.

Write to Gregor Stuart Hunter at gregor.hunter@wsj.com