My own self-worth should not be defined by the quantity or quality of what I produce.
Being productive in my free time has always been important to me. Well, maybe always is a stretch, but at least for the last decade I've been devoted to learning, creating and doing. Conversely, when I'm not being productive I can feel guilty for relaxing, consuming or just letting my mind wander.
Although quietly sitting not "doing" anything, may not meet the average definition of productivity, I think it can be exactly that. The processes of thinking creatively, exploring new ideas or reevaluating past experiences, are often very productive.
Maybe creativity can't even be separated from productivity. Whether the results are tangible or only thoughts that never leave my head, there is value in the experience.