DEV Community

Kamal Mustafa
Kamal Mustafa

Posted on

Python urldecode on command line

#python #bash

I have some logs that contain url with encoded characters and I need to extract some data out of the urls. Searching around lead me to this stackexchange's answer:-

alias urldecode='python3 -c "import sys, urllib.parse as ul;print(ul.unquote_plus(sys.argv[1]))"'
Enter fullscreen mode Exit fullscreen mode

Then you can use it as:-

urldecode '%22status%22%3A%22SUCCESS%22'
"status":"SUCCESS"
Enter fullscreen mode Exit fullscreen mode

Now that's perfect. But the data that I want to decode must be passed through unix pipe, like this:-

grep 'status: 504' /var/log/local0* | awk '{ print $10}' | xargs urldecode
Enter fullscreen mode Exit fullscreen mode

But this won't work, as xargs won't see the alias. So we alias xargs as well:-

alias xargs='xargs '
Enter fullscreen mode Exit fullscreen mode

But that however only work for the first line. How to make xargs execute for each line?

alias xargs='xargs -L 1 '
Enter fullscreen mode Exit fullscreen mode

Now we can run urldecode for each line we piped from grep!

Top comments (2)

Collapse
 
rrampage profile image
Raunak Ramakrishnan • • Edited

We can also do it without xargs by using sys.stdin to read from pipe
e.g:

yes 'Mozilla%2F5.0%20%28Macintosh%3B%20U%3B%20Intel%20Mac%20OS%20X%2010.6%3B%20en' |\
 head -n 10 |\
 python3 -c "import sys, urllib.parse as ul; [print(ul.unquote_plus(l), end='') for l in sys.stdin]"
Collapse
 
k4ml profile image
Kamal Mustafa •

Thanks for the tips!

Read next

drhyde profile image

Programmable tab completion with bash

David Cantrell -

ayush2390 profile image

9 cutting-edge open-source tools to build next-gen AI apps 🔮💡

Ayush Thakur -

cosmas_mandikonza_02df5e7 profile image

VisAssist: An Accessible Transcription Assistant for auditory impaired individuals.

Cosmas Mandikonza -

cloudnative_eng profile image

UV the game-changer package manager

Cloud Native Engineer -