You can e-mail [email protected] and ask for a firmware update of the CAC-1085. I don’t know if it will fix your problem.
On the DisplayPort side:
4K 120Hz 10bpc 4:2:0 (assuming 1188MHz) is possible without DSC using HBR3 link rate but not HBR2.
4:4:4 would require DSC but could work at HBR2 link rate.
GPU-Z in the advanced tab should show the link rate (8.1 Gbps) but I don’t know if Windows shows DSC anywhere. Is there an app that can show DPCD registers in Windows?
On the HDMI side:
It is not possible with a HDMI 2.0 connection without going to 8bpc 4:2:0.
HDMI 2.1 @ 6 Gbps without DSC is sufficient for 10bpc 4:2:0.
10bpc 4:4:4 requires DSC at 6 Gbps or non-DSC at 12 Gbps.
The FreeSync information menu shows some input connection info.
https://www.youtube.com/watch?v=dnUUAZY9HnQ
https://www.reddit.com/r/OLED/comments/kvxamh/ycbcr420_10b_tm_vs_rgb_10b_4l10/
4L10 is 4 lanes at 10 Gbps. HDMI 2.0 is shown as TM which is short for TMDS (3 lanes of 6 Gbps)
Maybe a custom timing of 4K 120Hz using CVT-RB would work better (1098 MHz). Then 4:4:4 can be done using 4L10 instead of 4L12.
Here’s another thread about the CAC-1085 and 4K120 10bpc:
https://insights.club-3d.com/thread/cac-1085-wont-work-at-444/