Plans to change a social club used by workers of an old city chocolate manufacturer have been met with their first criticism.