[Tut] [Collection] URL Decoding Methods

[Collection] URL Decoding Methods

URL encoding “is a method to encode information in a Uniform Resource Identifier (URI)”. It is also called Percent-encoding because percentage symbols are used to encode certain reserved characters:

!

#

$

%

&

'

(

)

*

+

,

/

:

;

=

?

@

[

]

%21

%23

%24

%25

%26

%27

%28

%29

%2A

%2B

%2C

%2F

%3A

%3B

%3D

%3F

%40

%5B

%5D

This article collects various ways to decode an URL encoded string. Let’s get started!

Python 2

$ alias urldecode='python -c "import sys, urllib as ul; \ print ul.unquote_plus(sys.argv[1])"' $ alias urlencode='python -c "import sys, urllib as ul; \ print ul.quote_plus(sys.argv[1])"'

Source

Here’s an example usage:

$ urldecode 'q+werty%3D%2F%3B'

q werty=/; $ urlencode 'q werty=/;'

q+werty%3D%2F%3B

Python 3

$ alias urldecode='python3 -c "import sys, urllib.parse as ul; \ print(ul.unquote_plus(sys.argv[1]))"' $ alias urlencode='python3 -c "import sys, urllib.parse as ul; \ print (ul.quote_plus(sys.argv[1]))"'

Here’s an example usage:

$ urldecode 'q+werty%3D%2F%3B'

q werty=/; $ urlencode 'q werty=/;'

q+werty%3D%2F%3B

Source

sed

$ sed 's@+@ @g;s@%@\\x@g' file | xargs -0 printf "%b"

Source

sed with echo -e

$ sed -e's/%\([0-9A-F][0-9A-F]\)/\\\\\x\1/g' file | xargs echo -e

Source

sed with alias

For convenience, you may want to use an alias:

$ alias urldecode='sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b"'

If you want to decode, you can now simply use:

$ echo "http%3A%2F%2Fwww" | urldecode

http://www

Source

Bash

input="http%3A%2F%2Fwww"

decoded=$(printf '%b' "${input//%/\\x}")

Source

To handle pluses (+) correctly, replace them with spaces using sed:

decoded=$(input=${input//+/ }; printf "${input//%/\\x}")

Bash + urlencode() + urldecode() Functions

urlencode() { # urlencode <string> local length="${#1}" for (( i = 0; i < length; i++ )); do local c="${1:i:1}" case $c in [a-zA-Z0-9.~_-]) printf "$c" ;; *) printf '%%%02X' "'$c" ;; esac done

} urldecode() { # urldecode <string> local url_encoded="${1//+/ }" printf '%b' "${url_encoded//%/\\x}"

}

Sources:

• StackOverflow
• https://github.com/sixarm/urldecode.sh

bash + xxd

urlencode() { local length="${#1}" for (( i = 0; i < length; i++ )); do local c="${1:i:1}" case $c in [a-zA-Z0-9.~_-]) printf "$c" ;; *) printf "$c" | xxd -p -c1 | while read x;do printf "%%%s" "$x";done esac

done

}

Sources:

• cdown’s gist
• Stackoverflow
• Another Source

PHP

$ echo oil+and+gas | php -r 'echo urldecode(fgets(STDIN));' // Or: php://stdin

oil and gas

Source

PHP Library

php -r 'echo urldecode("oil+and+gas");'

Source

Perl

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

Source

Perl to Process File

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

Source

awk

awk -niord '{printf RT?$0chr("0x"substr(RT,2)):$0}' RS=%..

Sources:

• anon‘s comment in StackOverflow article
• Source
• Using awk printf to urldecode text.

Python 2 urllib.unquote

The urllib.unquote is a special function in Python’s built-in standard library urllib that does what you need:

decoded_url=$(python2 -c 'import sys, urllib; print urllib.unquote(sys.argv[1])' "$encoded_url")

You can also use it to modify a file:

python2 -c 'import sys, urllib; print urllib.unquote(sys.stdin.read())' <file >file.new &&

mv -f file.new file

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Python 3 urllib.parse.unquote

If you run Python 3 on your system (like most people would), use the alternative function urllib.parse.unquote. To check your version, visit this article.

decoded_url=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.unquote(sys.argv[1]))' "$encoded_url")

Again, you can use the function to process a file as follows:

python3 -c 'import sys, urllib; print(urllib.parse.unquote(sys.stdin.read()))' <file >file.new &&

mv -f file.new file

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Perl URI::Escape

The URI::Escape solves the problem of URL decoding for Perl users.

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

You can use the function to process a file as follows:

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Perl One-Liner Without Installing Modules

$ perl -pe 's/\%(\w\w)/chr hex $1/ge'

Here’s a usage example:

$ echo '%21%22' | perl -pe 's/\%(\w\w)/chr hex $1/ge'

!"

Source: https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding

Bash Regex

$ function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; }

Now, you can use the function as a command like this:

$ urldecode https%3A%2F%2Fgoogle.com%2Fsearch%3Fq%3Durldecode%2Bbash

https://google.com/search?q=urldecode+bash

If you need to assign some variables, use this strategy:

$ x="http%3A%2F%2Fstackoverflow.com%2Fsearch%3Fq%3Durldecode%2Bbash"

$ y=$(urldecode "$x")

$ echo "$y"

http://stackoverflow.com/search?q=urldecode+bash

Source: https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell

GNU Awk

#!/usr/bin/awk -fn

@include "ord"

BEGIN { RS = "%.."

}

{ printf "%s", $0 if (RT != "") { printf "%s", chr("0x" substr(RT, 2)) }

}

Source: https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell

References

• Urlencode and urldecode from a command line
• Decoding URL encoding (percent encoding)
• How to decode URL-encoded string in shell?
• https://gist.github.com/cdown/1163649
• Python urldecode on command line

https://www.sickgaming.net/blog/2020/08/...g-methods/