The method to calculate the buckling loads and modes of a frame is explained in this lecture. The basic idea is that you assume a buckling mode of a frame, and back-calculate the corresponding buckling load. If you do this for a couple of buckling modes, you can assess which one of the buckling loads is the lowest, and that one will be the critical load of the frame, and the corresponding buckling mode will be the critical mode.

The Lecture Slides

There is an error on page 5 of the lecture notes. The stability equation should be EIw”+Pw = 0.