Two million people are believed to have died under the Khmer Rouge
"These rules will ensure us fair and transparent trials," co-prosecutor Robert Petit told reporters.
The deal was reached in a week-long meeting, after a delay of more than six months because of disagreements between local and UN-appointed legal officials.
The tribunal, which has a budget of $56.3m, is expected to run for three years, starting trials in 2008.
Mr Petit said the rules had been agreed unanimously by the panel.
"Now that the rules are adopted, we can move forward," he said.
A UN appointed judge recently said the investigation phase, including arrests of potential defendants, could begin within weeks of the rules being adopted, the Associated Press reported.
A meeting in November last year over ended in disarray - and the future of the process looked bleak.
But this time the mood was cordial.
Earlier officials told the BBC that there were no serious disagreements during the meeting.
Many of the legal officials have had little to do during the long hiatus caused by the dispute.
BBC correspondent Guy De Launey said the prosecutors have continued to compile evidence - and they are now ready to give their files to the investigating judges.
It means that there should soon be official confirmation of which former Khmer Rouge leaders will be charged - and with what crimes.
As many as two million people are thought to have died during the four years of Khmer Rouge government in the late 1970s.
The movement's former leader, Pol Pot, died nine years ago.But the former head of state, Khieu Samphan, and Foreign Minister Leng Sary have both been living freely in Cambodia.