Cached means used, not used as a disk cache. Notice your numbers don't add up. Cached = "in use" + modified. Available = standby + free. Use Resource Monitor (that little button at the bottom of task manager) to get a more detailed usage.
Where you'll see more performance without using the memory is when free is small. Standby is the OS's guesses as to what will end up in Cached. When it guesses right, you get better performance. This is closer to a "cache" in the way I think you are thinking of it. So as "free" shrinks, the OS shrinks standby to ensure you don't run out. This leads to fewer standby hits, making for a worse performance.
That's why in Windows (up to 7, have't tested with 8 and 10) Windows will if you have a page file page used memory, making the system slow as dirt. Leave your computer on overnight, and nearly everything will be paged, to maximize "Free" and "Standby", optimizing the system for new applications opened, and greatly punishing a user who locks his computer overnight with Chrome open with 20 tabs.
That's why someone with 16G should disable a paging file in Windows, for better performance.