A quick Linux idea.
I used zram on the raspberry pi with good performance. So I wonder. Similar to cpulimit on Linux. Would it be possible to use memory compression per process?
Then with some smart decision making algorithm. You could keep the performance of the whole system and ?still reduce memory load.