UPDATE: I figured out how to install this in VS code.
You can download git bash online and it will be installed on your local computer. In VS Code, you can then Ctrl + Shift + P and type "Open User Settings". In the right pane, add this setting:

This will replace the default built-in terminal in VSCode with the git bash CLI client that was just installed in the location in the screenshot above. Now that your local file system is installed in VScode, you can do:
ssh username@hostname -p <port>
source
share