State Street Corp. agreed to buy Brown Brothers Harriman & Co.’s investor-services unit for $3.5 billion in cash, a deal that expands the custody bank’s core business serving investment firms.

Brown Brothers’s BBH Investor Services business performs accounting and administrative tasks for asset managers and pensions, overseeing $5.4 trillion in assets under custody. State Street chief executive Ron O’Hanley said that adding those assets to its own $31.9 trillion will lift State Street above rival Bank of New York Mellon Corp. as the money-management industry’s biggest servicing firm.

The agreement is State Street’s first significant acquisition since it bought financial-data firm Charles River System Inc. three years ago. Unlike that deal, which gave State Street a toehold into their clients’ trading floors, Tuesday’s transaction aims to expand its turf in many of those same customers’ back offices.

State Street and its rivals have long struggled to jump-start revenue growth from asset-servicing businesses as their clients, facing their own pressures to lower costs, push down on the fees they pay for administrative functions like tracking the value of their assets. Getting bigger can help counter those moves.

“There’s a fair amount of economies of scale in this business,” Mr. O’Hanley said in an interview.

State Street said Tuesday the acquisition would add to its per-share earnings within a year and lift its targeted pretax profit margin. Within three years, State Street said, the acquisition will trim expenses by $260 million.

The bank’s shares fell 3.8% to $89.32 in midday trading on Tuesday. They have climbed 36% in the past year.

Brown Brothers Harriman will continue owning and operating independently its separate private-banking and investment-management businesses after the transaction closes, the companies said.

State Street said the deal will support its services through data-connectivity tools and by expanding in non-U.S. markets such as Japan, Luxembourg, Ireland and Latin America.

The Brown Brothers’s business had revenue of $1.4 billion last year, of which about 70% came from international clients.

State Street said it expects to fund the transaction primarily through the issuance of common equity, the suspension of common share repurchases before resuming in the second quarter of 2022 and cash on hand.

BBH Investor Services employees will join State Street, and its senior management team will become State Street executives, the companies said. Seán Páircéir, currently partner and global head of investor services at BBH, will join State Street’s management committee, they added.

Write to Justin Baer at justin.baer@wsj.com and Dave Sebastian at dave.sebastian@wsj.com