I'd say this hardly depends on the code you are compiling.
Whatever you choose there, will end up as flag for GCC, so following GCC's manpage:
-O the compiler tries to reduce code size and execution time, without performing any optimizations that take a great deal of compilation time.
-O2 Optimize even more. GCC performs nearly all supported optimizations that do not involve a space-speed tradeoff. As compared to -O, this option increases both compilation time and the performance of the generated code.
-O3 Optimize yet more.
-O3 turns on all optimizations specified by
-O2 and also turns on the
You can run
gcc -O<1-3> -Q --help=optimizers
to see the set of activated optimization methods for each optimization level.
The more aggressive they are, the bigger the risk that the compiler will screw something up (should not happen, and I don't know how likely something will happen, but if you want to be on the safe side, I suggest you follow apt-build's recommendation not to use