Getting proxy from file Error cURL - (5) Unsupported proxy syntax
Today for a project, was trying to hit an endpoint URL with post request via a lit of proxy. 1st was trying with Burp Repeater , but then realise, there is not direct function exists to add 1000s of upstream proxy in a shot. So, thought of settling with good ol' cURL.
So, the syntax to use proxy with curl request is like
-x, --proxy [protocol://]host[:port]
curl "URL" --proxy PROXY_IP:PORT
or
curl "URL" --socks5 socks5_ip:port
So, to use this easiest option created a loop to take proxy from a text file and execute command per line
for line in $(cat proxy.txt); do curl COMMAND --proxy "$line"; done
But - WTF , started getting error like
curl: (5) Unsupported proxy syntax in '$line'
curl: (5) Unsupported proxy syntax in '$line'
curl: (5) Unsupported proxy syntax in '$line'
curl: (5) Unsupported proxy syntax in '$line'
curl: (5) Unsupported proxy syntax in '$line'
curl: (5) Unsupported proxy syntax in '$line'
Tried with single line manally and worked well. But, when fetching from proxy file, it is not working.
After a bit of research, found, the culprit , it is \r in txt file, the input file has carriage return characters at the end of each line. So, when the command taking one line from the txt file , it is returning 127.0.0.1:7070\r rather than 127.0.0.1:7070 . What a loose of time !!!
So, formatted the command line in different way. Used tr to remove '\r' from the end of each line and after that it went fine.
while read line; do curl -p "URL" --proxy "$line" --connect-timeout 2; done < <( tr -d '\r' < proxy.txt )