Fashion group criticised by shareholder groups and pay campaigners