All major OSes compress memory at this point. I'd be curious to see how OoM acts on macOS -- I've seen it on Windows [NT4] and Linux over the years. I've not seen an OS handle it as well as NT does/did.
zswap won't compress pages until they're swapped to disk, so essentially only once you've run out of memory to use and have begun experiencing the horrid performance degradation that comes from swapping to disk. you can use zram instead and put swappiness to a high value so the kernel aggressively swaps pages to the fake swap device made by zram, but then you lose the ability to have on-disk compressed swap pages.
i'm surprised that more work hasn't been done to fix this because memory management on linux is clearly lagging behind and affects a lot more than the minority of people using desktop linux systems, every android phone ships with 1.5x-2x more RAM than the equivalent iphone and performs worse when a bunch of apps are running in the background because of the poor memory management.
Linux zram seems effective for marking some memory as compressed swap. Alas it doesn't play well with zswap, reportedly, which lets real swap drives also be compressed. But should be useful in many scenarios to quickly switch between compressed memories in a wide number of cases.
As someone else mentioned, for many cases this will be a performance increase, not decrease. Compressing infrequently used memory frees up more memory to be used for filesystem caching among other things.
All major OSes compress memory at this point. I'd be curious to see how OoM acts on macOS -- I've seen it on Windows [NT4] and Linux over the years. I've not seen an OS handle it as well as NT does/did.