There are a lot, the most mainstream are going to fall under "socialism". But to give you a specific answer, check out "Alternatives to Capitalism: Proposals for a Democratic Economy". It's a debate between Robin Hahnel and Erik Olin Wright around their personal ideas.
More broadly speaking, any number of systems could compensate people for innovation and problem solving (note I don't think competition should be rewarded in its own right, and risk taking isn't necessary in many systems) by providing them an elevated quality of life.
The main difference to our capitalist system today is that rather than trying to accumulate as much personal capital as possible, value created from innovation would only go to the individual as needed to incentivize them. The rest would go to improving the world and lives of the people around them.
Of course this requires the individual have some agency in that improvement, which we find lacking in the Soviet Union or other failed communist states.
More broadly speaking, any number of systems could compensate people for innovation and problem solving (note I don't think competition should be rewarded in its own right, and risk taking isn't necessary in many systems) by providing them an elevated quality of life. The main difference to our capitalist system today is that rather than trying to accumulate as much personal capital as possible, value created from innovation would only go to the individual as needed to incentivize them. The rest would go to improving the world and lives of the people around them. Of course this requires the individual have some agency in that improvement, which we find lacking in the Soviet Union or other failed communist states.