Did you have the address space to the virtual network gateway? When you created that VPN connection and identified the subnets in your on prem environment, Azure auto creates routes as system defaults. Have you validated the data packets arriving at the FW?
You should not need any manual routes for the Azure=> onprem to work. Only needed if you want to force the default route via onprem. Since the other direction works, return packets are routed correctly. Perhaps check your onprem firewall policies and packet capture on the onprem firewall.
Make sure route propagation isn’t disabled on Ur vnet…you can also over ride this by adding a UDR to the subnet and force the VM to point to the gateway for Ur range on prem