[Tut] [Collection] URL Decoding Methods

[xSicKxBot]

[Collection] URL Decoding Methods

URL encoding “is a method to encode information in a Uniform Resource Identifier (URI)”. It is also called Percent-encoding because percentage symbols are used to encode certain reserved characters:

Code:

!

|

Code:

#

|

Code:

$

|

Code:

%

|

Code:

&

|

Code:

'

|

Code:

(

|

Code:

)

|

Code:

*

|

Code:

+

|

Code:

,

|

Code:

/

|

Code:

:

|

Code:

;

|

Code:

=

|

Code:

?

|

Code:

@

|

Code:

[

|

Code:

]

Code:

%21

|

Code:

%23

|

Code:

%24

|

Code:

%25

|

Code:

%26

|

Code:

%27

|

Code:

%28

|

Code:

%29

|

Code:

%2A

|

Code:

%2B

|

Code:

%2C

|

Code:

%2F

|

Code:

%3A

|

Code:

%3B

|

Code:

%3D

|

Code:

%3F

|

Code:

%40

|

Code:

%5B

|

Code:

%5D

This article collects various ways to decode an URL encoded string. Let’s get started!

Python 2

Code:

$ alias urldecode='python -c "import sys, urllib as ul; \ print ul.unquote_plus(sys.argv[1])"' $ alias urlencode='python -c "import sys, urllib as ul; \ print ul.quote_plus(sys.argv[1])"'

Source

Here’s an example usage:

Code:

$ urldecode 'q+werty%3D%2F%3B' q werty=/; $ urlencode 'q werty=/;' q+werty%3D%2F%3B

Python 3

Code:

$ alias urldecode='python3 -c "import sys, urllib.parse as ul; \ print(ul.unquote_plus(sys.argv[1]))"' $ alias urlencode='python3 -c "import sys, urllib.parse as ul; \ print (ul.quote_plus(sys.argv[1]))"'

Here’s an example usage:

Code:

$ urldecode 'q+werty%3D%2F%3B' q werty=/; $ urlencode 'q werty=/;' q+werty%3D%2F%3B

Source

sed

Code:

$ sed 's@+@ @g;s@%@\\x@g' file | xargs -0 printf "%b"

Source

sed with echo -e

Code:

$ sed -e's/%\([0-9A-F][0-9A-F]\)/\\\\\x\1/g' file | xargs echo -e

Source

sed with alias

For convenience, you may want to use an alias:

Code:

$ alias urldecode='sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b"'

If you want to decode, you can now simply use:

Code:

$ echo "http%3A%2F%2Fwww" | urldecode http://www

Source

Bash

Code:

input="http%3A%2F%2Fwww" decoded=$(printf '%b' "${input//%/\\x}")

Source

To handle pluses (

Code:

+

) correctly, replace them with spaces using

Code:

sed

:

Code:

decoded=$(input=${input//+/ }; printf "${input//%/\\x}")

Bash + urlencode() + urldecode() Functions

Code:

urlencode() { # urlencode <string> local length="${#1}" for (( i = 0; i < length; i++ )); do local c="${1:i:1}" case $c in [a-zA-Z0-9.~_-]) printf "$c" ;; *) printf '%%%02X' "'$c" ;; esac done } urldecode() { # urldecode <string> local url_encoded="${1//+/ }" printf '%b' "${url_encoded//%/\\x}" }

Sources:

• StackOverflow
  
  
• https://github.com/sixarm/urldecode.sh
  

bash + xxd

Code:

urlencode() { local length="${#1}" for (( i = 0; i < length; i++ )); do local c="${1:i:1}" case $c in [a-zA-Z0-9.~_-]) printf "$c" ;; *) printf "$c" | xxd -p -c1 | while read x;do printf "%%%s" "$x";done esac done }

Sources:

• cdown’s gist
  
  
• Stackoverflow
  
  
• Another Source
  

PHP

Code:

$ echo oil+and+gas | php -r 'echo urldecode(fgets(STDIN));' // Or: php://stdin oil and gas

Source

PHP Library

Code:

php -r 'echo urldecode("oil+and+gas");'

Source

Perl

Code:

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

Source

Perl to Process File

Code:

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

Source

awk

Code:

awk -niord '{printf RT?$0chr("0x"substr(RT,2)):$0}' RS=%..

Sources:

• anon‘s comment in StackOverflow article
  
  
• Source
  
  
• Using awk printf to urldecode text.
  

Python 2 urllib.unquote

The

Code:

urllib.unquote

is a special function in Python’s built-in standard library

Code:

urllib

that does what you need:

Code:

decoded_url=$(python2 -c 'import sys, urllib; print urllib.unquote(sys.argv[1])' "$encoded_url")

You can also use it to modify a file:

Code:

python2 -c 'import sys, urllib; print urllib.unquote(sys.stdin.read())' <file >file.new && mv -f file.new file

Source: https://unix.stackexchange.com/questions...t-encoding

Python 3 urllib.parse.unquote

If you run Python 3 on your system (like most people would), use the alternative function

Code:

urllib.parse.unquote

. To check your version, visit this article.

Code:

decoded_url=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.unquote(sys.argv[1]))' "$encoded_url")

Again, you can use the function to process a file as follows:

Code:

python3 -c 'import sys, urllib; print(urllib.parse.unquote(sys.stdin.read()))' <file >file.new && mv -f file.new file

Source: https://unix.stackexchange.com/questions...t-encoding

Perl URI::Escape

The

Code:

URI::Escape

solves the problem of URL decoding for Perl users.

Code:

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

You can use the function to process a file as follows:

Code:

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

Source: https://unix.stackexchange.com/questions...t-encoding

Perl One-Liner Without Installing Modules

Code:

$ perl -pe 's/\%(\w\w)/chr hex $1/ge'

Here’s a usage example:

Code:

$ echo '%21%22' | perl -pe 's/\%(\w\w)/chr hex $1/ge' !"

Source: https://unix.stackexchange.com/questions...t-encoding

Bash Regex

Code:

$ function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; }

Now, you can use the function as a command like this:

Code:

$ urldecode https%3A%2F%2Fgoogle.com%2Fsearch%3Fq%3Durldecode%2Bbash https://google.com/search?q=urldecode+bash

If you need to assign some variables, use this strategy:

Code:

$ x="http%3A%2F%2Fstackoverflow.com%2Fsearch%3Fq%3Durldecode%2Bbash" $ y=$(urldecode "$x") $ echo "$y" http://stackoverflow.com/search?q=urldecode+bash

Source: https://stackoverflow.com/questions/6250...g-in-shell

GNU Awk

Code:

#!/usr/bin/awk -fn @include "ord" BEGIN { RS = "%.." } { printf "%s", $0 if (RT != "") { printf "%s", chr("0x" substr(RT, 2)) } }

Source: https://stackoverflow.com/questions/6250...g-in-shell

References

• Urlencode and urldecode from a command line
  
  
• Decoding URL encoding (percent encoding)
  
  
• How to decode URL-encoded string in shell?
  
  
• https://gist.github.com/cdown/1163649
  
  
• Python urldecode on command line