Fewer workers typically means less production of goods and services, and this worries the leaders who met in Jackson Hole.