I didn't use an Ergo, but I used a Manduca which is quite similar. I used it all the way through my pregnancy, wearing my toddler on my back (on the front got impossible after not too long, bump in the way) - actually it was quite nice wearing it because I had the padded waistband under my bump, so the toddler on my back lifted my bump a little so it was quite nicely supported!
HTH