Hilary Duff is Struggling with This After Giving Birth to Daughter Mae