At the beginning of the movie, during the 1920s, Building & Loans were huge. But by the end of the movie, right after World War 2, in real life, most Building & Loans had already gone out of business, or had been converted into Savings & Loans.
One way or another, if the movie had continued, the Bailey Brothers Building and Loan would have soon been gone.
Building & Loans were often extremely small like the Bailey Brothers Building and Loan. One study found the median number of members in Building & Loan Associations in Newark, NJ in 1930, for example, was only 450 people.
In the movie, George Bailey himself called his Building & Loan, a “cheap penny-ante Building and Loan” and “measly one-horse institution” and later refers to it helping (only) 100 homeowners.