Sounds more like an EDID problem. I don't know how to debug that on Windows, but you can check that on Linux, search for the EDID blob on the 'Net, and install it on Windows.
Eg. for one of my monitors I have this bit in .xsessionrc:
xrandr --newmode "1600x1200"x60.0 162.00 1600 1664 1856 2160 1200 1201 1204 1250 +hsync +vsync
xrandr --addmode HDMI-A-1-5 1600x1200x60.0
xrandr --output HDMI-A-1-5 --mode 1600x1200x60.0
xrandr --output HDMI-A-1-5 --rotate left
xrandr --output HDMI-A-1-5 --left-of DP-1
I just realized this would be better put into /etc/x11/xorg.conf like we used to in the previous millennium -- that'd let saved layout apply automatically -- but hey, it works.
On the other hand, Windows requires raw EDID rather than parsed modelines.