By Matt Jarzemsky and Chris Dieterich

When midsize stocks retreated as their large and small counterparts rose Monday, traders scratched their heads about the unusual activity. The culprit, traders later surmised, was heavy selling in two exchange-traded funds--and the incident underscored the rising influence such funds have on the market.

"The tail was wagging the dog. The mid-cap weakness was driven by the ETF flow," said Jenkins Marshall, managing director of equities sales and trading at Knight Capital Group Inc.

Despite gains elsewhere in the market, the Standard & Poor's index of 400 mid-capitalization stocks slumped 0.5% Monday. That compared with a 0.3% gain in the S&P 500, which represents larger companies, and a 0.4% advance in S&P's index of 600 small-capitalization stocks. It was just the fifth session in the past five years when the S&P's large- and small-cap indexes have risen while the mid-cap benchmark fell, according to FactSet. The last time that occurred was Jan. 20.

The Standard & Poor's index of 400 mid-cap stocks struggled out of the gate after Monday's opening bell, swinging between gains and losses before heading firmly into negative territory around noon EDT. The performance mirrored two exchange-traded funds that sold off on heavy volume throughout the day, with each setting a record for the number of shares traded in one session.

The extraordinary volume in the two funds, which included a handful of single transactions worth close to $150 million each, convinced some market observers they had spotted a case where large ETF trades were influencing the underlying securities, as opposed to the other way around.

ETFs normally trade in tandem with their component stocks; sophisticated traders watch for discrepancies between a fund's value and that of its components and seek to profit by closing the gaps.

By contrast, market observers said Monday's weakness in midcap stocks was likely due at least in part to brokers selling S&P 400 stocks as they handled big sales of two ETFs: the iShares S&P 400 MidCap Index Fund (IJH), which tracks the index, and the ProShares Ultra MidCap 400 (MVV), which is designed to double the benchmark's daily move. Brokers who make a market in ETFs stand ready to take the other side of clients' trades, and they often buy or sell the underlying securities to offset positions.

A handful of afternoon trades underlined the pressure the selling put on the market. At 2:02 p.m. EDT, a trade for 1.5 million shares of the iShares S&P 400 MidCap Index Fund hit the tape--the single transaction exceeding the fund's average daily volume over the 30 days before Monday--at $98.52, 0.2% lower than the ETF's previous closing price. Three more trades of at least 1.5 million shares hit the tape within the hour, each helping to keep the ETF in negative territory. The largest transaction of the day saw $163 million change hands.

In another sign that the mid-cap weakness was tied to the ETF activity, trading volume in S&P 400 stocks Monday was about $2.8 billion higher than the 20-day average. That figure nearly matched the day's dollar volume in the two ETFs, according to Credit Suisse Trading Strategy.

"It seems like somebody's selling mid- and buying large-caps," Frank Ingarra, head trader at NorthCoast Asset Management, said as the activity unfolded. Mr. Ingarra said a large institutional investor was likely responsible for at least part of the volume, though that isn't certain because the identity of who made the trades isn't public.

Representatives for ProShares and iShares, a unit of BlackRock Inc. (BLK), weren't immediately able to provide a comment.

The action shows how ETFs, well-known to individual investors looking for equity-index exposure at a low cost, can also play a role in large, market-moving buying and selling.

"To me, the most amazing thing in all this is it just shows the dominance of ETFs as the asset class of choice to get your exposure to equities," Knight's Mr. Marshall said.

In another indication of the activity rippling through the market, the iShares mid-cap fund lost $899.3 million worth of assets Monday, sharply more than any other ETF, according to XTF, a market-data provider. Some of the outflows may have resulted from ETF market makers trading in units of the product for the underlying shares, a step brokers use to manage their risk after facilitating an ETF sale.

Meanwhile, volume in individual stocks in the index surged. Executive-search firm Korn/Ferry International (KFY) and railroad operator Genesee & Wyoming Inc. (GWR) were among stocks in the mid-cap fund whose volume was more than eight times the 30-day average.

Not all Monday's action can be attributed to an ETF-influenced anomaly. Monday's trend also shines a light on the relative underperformance of mid-cap stocks in recent weeks. The S&P 400 MidCap index, which tracks stocks such as Vertex Pharmaceuticals Inc. (VRTX) and Equinix Inc. (EQIX), has trailed both its small- and large-cap counterparts over the past month.

"You definitely get a sense that there's a rotation" away from mid-cap stocks, said Rick Fier, director of equity trading at Conifer Securities.

Write to Matt Jarzemsky at matthew.jarzemsky@dowjones.com.

Subscribe to WSJ: http://online.wsj.com?mod=djnwires