Most likely, the problem lies in something else than in the drivers or the video card / monitor. I would conduct a full diagnostic of all components.

Afaik, I can’t see where it says the GTX1050 TI is supported by the CAC-1085 on their product site?

Also the DP ports on the GTX1050 TI are DP1.2 and DP1.3/1.4 ready. Try to upgrade the firmware of the card?

https://www.nvidia.com/en-sg/geforce/products/10series/geforce-gtx-1050/

I had already updated the firmware.  It does have DP1.4 but no DSC so the max I can get out of it is 4k@120hz HDR in 4:4:4 which I guess is good enough.  I had to force a custom resolution in the nvidia control panel though.  It is weird the colors seem darker compared to when I use a regular HDMI cable though. even though they have the same settings for both ports on my tv.

DP 1.2 has more bandwidth than HDMI 2.0 but doesn’t support 4:2:0. Depending on the display and the adapter, DP 1.2 may be able to do the following 4K modes (assuming the display can do HDMI 2.1; assuming CVT-RB timing/not HDMI timing; assuming 3840×2160):

• 60Hz 10bpc 4:4:4
• 80Hz 8bpc 4:4:4
• 95Hz 10bpc 4:2:2
• 110Hz 8bpc 4:2:2

The Pascal series of Nvidia cards should be able to support DisplayPort 1.4 but they do not support DSC which the CAC-1085 uses to get some HDMI 2.1 modes. Without DSC, you should be able to get the following 4K timings (give or take a few Hz):

• 95Hz 10bpc 4:4:4
• 115Hz 8bpc 4:4:4
• 140Hz 10bpc 4:2:2
• 170Hz 8bpc 4:2:2
• 180Hz 10bpc 4:2:0
• 220Hz 8bpc 4:2:0

Notes:

• 10bpc is required for HDR.
• 4:4:4 uses same bandwidth as RGB.
• The adapter or display might not support some modes.

If Nvidia drivers are crashing then maybe that’s an issue for Nvidia to solve – after you make sure you have the latest drivers.

Try Linux to see if the adapter behaves better in a different OS.