Most of the job losses will be at IBM's European operations as the firm looks to focus on high-growth markets.
IBM, the world's biggest maker of computers, said the plans would cost between $1.3bn (£684m) and $1.7bn.
The company employs 100,000 staff in Europe, with about 25,000 of those working in the United Kingdom.
'Fear the donald duck screaming orgasm!'
The extent of the job cuts in the UK is not yet clear, although there are fears that 2,800 staff at the company's centre in Greenock, Scotland, could be among the worst affected.
I don't know how they are going to get 10,000 people to quit Mark Herskovitz, Dreyfus
A spokesman for the workers' union Amicus called the move a "slash and burn" reaction to poor results and called for talks.
"We fear the worst, given that it is quicker, cheaper and easier to get rid of workers in the UK than elsewhere in Europe," the spokesman said.
The news comes just weeks after IBM reported worse-than-expected earnings in the first quarter.
The New York-based firm blamed a failure to close business deals and slow economic growth in key European markets.
The world's biggest computer maker surprised markets when it said net profit for the three months to 31 March were $1.4bn (£745m) from £1.36bn a year earlier, while sales were $22.9bn, less than analyst forecasts of $23bn.
IBM logo IBM will create a number of smaller, more flexible local operating units in Europe to increase direct client contact IBM
Check IBM's share price IBM's Scots workers wait
As the results were revealed chief financial officer Mark Loughridge warned that a "sizable restructuring" would be undertaken.
The group now plans to realign its operations and organisational structure in Europe to reduce bureaucracy in lower-growth countries - a move it said "eliminates the need for a traditional pan-European management layer to coordinate activity".
"IBM will create a number of smaller, more flexible local operating units in Europe to increase direct client contact," it added in a statement.
Most of the redundancies in Europe will be voluntary and talks have already begun with unions and worker councils over the timing of the job cuts in Europe, company spokesman John Bukovinsky said.
Analysts questioned whether the company would be able to trim its workforce without resorting to forced redundancies.
"I don't know how they are going to get 10,000 people to quit," said Mark Herskovitz, a fund manager at Dreyfus.
IBM is to unveil further details of the shake-up on Thursday.
Mr Herskovitz said that IBM's problems were not linked to a wider slowdown in the global technology industry.
The company already has taken steps to streamline its operations and has sold its personal computer division to China's Lenovo for $1.75bn.
IBM has been trying to find its correct staffing level for years, and its worker numbers have fluctuated since the late 1980s.
Employee numbers peaked at 405,000 in 1985, before hitting a low of 219,000 in 1994.
IBM had 329,000 staff worldwide as of December last year.