Wall Street professionals tell everyday investors what stocks to buy. But now they have to follow some of these amateurs for signs of where the market is headed next.

Venerable institutions Goldman Sachs Group Inc. and Morgan Stanley are tracking the retail trading frenzy, and hedge funds in New York and London have employees combing through the internet forum of Reddit, Twitter or chat startup Discord in search of trading opportunities. They turned to these sources following a period of market mayhem dominated by amateurs on Reddit’s WallStreetBets and the Robinhood Markets Inc. trading app who collectively boosted the shares of GameStop Corp. and other companies that had fallen out of Wall Street’s favor.

“It’s more art than science because it’s uncharted territory,” said Simeon Siegel, a BMO Capital Markets analyst who has spent his career covering retail companies.

One analyst who turned to WallStreetBets this year for insight was Priyesh Mehta, 26, who advises Cayman Islands-based hedge fund Bovell Global Macro Fund. He said he never considered that a group of traders could band together on online forums with the aim of jolting prices, but as GameStop’s shares rose in January he downloaded the Reddit app on his phone, made an account and joined the forum.

Mr. Mehta keeps track of the Reddit thread WallStreetBets from his home in London.

Photo: Francesca Jones for The Wall Street Journal

For the next few weeks, he spent hours on the platform, familiarizing himself with its design. Mr. Mehta quickly learned that the place to pay attention to was the daily thread of people’s trades. He began to recognize names of accounts that posted frequently, an indication that those users may have more sway with the group. Mr. Mehta still monitors the forum and warns Bovell’s fund manager of highly mentioned stocks that could see volatility. He is also looking for potential stocks that the firm could bet against.

“I never thought that retail traders would actually gain this much power in the market,” he said.

This isn’t the first time everyday investors grabbed the attention of Wall Street. Day traders first bloomed during the dot-com boom of the 1990s, and by 1999 the Securities and Exchange Commission chair estimated there were fewer than 7,000 of them.

Their numbers appear to be bigger now. During the first half of 2021, individual investors opened about 10 million new brokerage accounts, according to JMP Securities. Stock-trading app Robinhood said that it had 22.5 million accounts as of June 30, up from 18 million on March 30. This month, shares of Robinhood rallied 50% in a single day, driven by a surge of retail bets.

The surge has boosted the influence of WallStreetBets, the Reddit forum that powered a rise this year in the stocks of troubled companies such as GameStop and AMC Entertainment Holdings Inc. At the start of 2018, it had 200,000 subscribers on Reddit—a number that would quadruple by the start of 2020. It now has nearly 11 million.

The activities of these everyday investors is now under close scrutiny at New York-based Cindicator Capital, which has two traders spending the bulk of their days looking for trading opportunities on Reddit and Twitter, according to Nodari Kolmakhidze, the firm’s chief financial officer. In February, Cindicator even posted a job for a sentiment trader that required a refined taste for memes and membership on WallStreetBets that spanned at least a year, before deciding not to fill the post.

The firm uses a mix of survey data and artificial intelligence to track sentiment. Mr. Kolmakhidze said he first noticed GameStop being mentioned in August 2020 on a Discord chat the firm created for retail traders, but that he paid it little attention at the time. Mr. Kolmakhidze said he now views everyday investors as a force likely to shift markets for the foreseeable future, creating an opportunity to bet for or against assets favored by the crowd.

“When you see these trades, you are like ‘wow, I could be there,’” he said.

The GameStop frenzy put the spotlight on a growing group of investors who seek and share trading information on social media platforms like YouTube and TikTok. Three investors explain how these online communities are helping them chase the market. Photo illustration: Adam Falk/The Wall Street Journal The Wall Street Journal Interactive Edition

Write to Caitlin Ostroff at caitlin.ostroff@wsj.com and Paul Vigna at paul.vigna@wsj.com