"The Dosadi Experiment" by Frank Herbert
The Dosadi Experiment was first published in 1978. It is the second book (and the third story) in Herbert's ConSentiency series. It is also the sequel to the book Whipping Star, also reviewed here.
The Dosadi Experiment is a terrible book, and is one of the worst things that Frank Herbert ever wrote. The plot is a mess, the characters are wooden and uninteresting (even for Herbert), and none of the ideas in this book are carried through to their logical conclusions. It is a book full of annoying aphorisms that might make sense if you were really, really stoned, and dramatic tension is almost entirely absent from this novel.
Apparently a frog-like race, the Gowachin, have created a large-scale social experiment on the planet Dosadi, involving both humans and Gowachins transplanted to the surface of that planet. Following his discovery of this "Dosadi Experiment" an agent of the Bureau of Sabotage (or BuSab, for short) journeys to Dosadi, where he makes a universe-shattering discovery.
The only trouble with all of this is that BuSab's functions are never clear, the "threat" embodied in Dosadi is likewise unclear, and the "court scene" which concludes this book makes about as much sense as the court scene which concluded Whipping Star.
There's much better sci-fi out there, and there are much better books by Frank Herbert.