You should always ask the host first. If you are the host, you should let others know when it is appropriate to take what is left over. Typically it would be at the end of the break session before they refresh the table, or at the end of the day or meeting.
You also sound like you might be a Utilitarian in your attitude and beliefs, meaning that you don’t want to see anything go to waste. Wastefulness to a Utilitarian drives them crazy!
To learn more about CHUVA group’s Rules Of Engagement…the business meal program, click here.