pz
Ever wished to use Python in Bash? Would you choose the Python syntax over sed
, awk
, ...? Should you exactly know what command would you use in Python, but you end up querying man
again and again, read further. The utility allows you to pythonize the shell: to pipe arbitrary contents through pz
, loaded with your tiny Python script.
How? Simply meddle with the s
variable. Example: appending '.com' to every line.
$ echo -e "example\nwikipedia" | pz 's += ".com"'
example.com
wikipedia.com
- Installation
- Examples
- Extract a substring
- Prepend to every line in a stream
- Converting to uppercase
- Parsing numbers
- Find out all URLs in a text
- Sum numbers
- Keep unique lines
- Counting words
- Fetching web content
- Handling nested quotes
- Computing factorial
- Read CSV
- Generate random number
- Average a stream value
- Multiline statements
- Simple progress bar
- Docs
- Scope variables
s
– current linen
– current line converted to anint
(orfloat
) if possibleb
– current line as a byte-stringcount
– current line numbertext
– whole text, all lines togetherlines
– list of lines so far processednumbers
– list of numbers so far processedskip
linei
,S
,L
,D
,C
– other global variables
- Auto-import
- Output
- CLI flags
- Scope variables
Installation
Install with a single command from PyPi.
pip3 install pz
Or download and launch the pz
file from here.
Examples
How does your data look when pythonized via pz
? Which Bash programs may the utility substitute?
Extract a substring
Just use the [:]
notation.
echo "hello world" | pz s[6:] # world
Note that suppressing quotes around the argument may not work (Zsh) or lead to an unexpected behaviour: touch s1 && echo "hello" | pz s[1]
→ Exception: <class 'NameError'>
. Use echo "hello" | pz 's[1]'
instead.
Prepend to every line in a stream
We prepend the length of the line.
# let's use the f-string `--format` flag
tail -f /var/log/syslog | pz -f '{len(s)}: {s}'
# or do it the long way, explicitly setting the `s` variable
tail -f /var/log/syslog | pz 's = str(len(s)) + ": " + s'
Converting to uppercase
Replacing | tr '[:upper:]' '[:lower:]'
.
echo "HELLO" | pz s.lower # "hello"
Parsing numbers
Replacing cut
. Note you can chain multiple pz
calls. Split by a comma ',
', then use n
to access the line converted to a number.
echo "hello,5" | pz 's.split(",")[1]' | pz n+7 # 12
Find out all URLs in a text
Replacing sed
. We know that all functions from the re
library are already included, ex: "findall".
# either use the `--findall` flag
pz --findall "(https?://[^\s]+)" < file.log
# or expand the full command to which is the `--findall` flag equivalent
pz "findall(r'(https?://[^\s]+)', s)" < file.log
If chained, you can open all the URLs in the current web browser. Note that the function webbrowser.open
gets auto-imported from the standard library.
pz --findall "(https?://[^\s]+)" < file.log | pz webbrowser.open
Sum numbers
Replacing | awk '{count+=$1} END{print count}'
or | paste -sd+ | bc
. Just use sum
in the --end
clause.
# internally changed to --end `s = sum(numbers)`
echo -e "1\n2\n3\n4" | pz --end sum # 10
Keep unique lines
Replacing | sort | uniq
makes little sense, but the demonstration gives you the idea. We initialize a set c
(like a collection). When processing a line, skip
is set to True
if already seen.
$ echo -e "1\n2\n2\n3" | pz "skip = s in c; c.add(s)" --setup "c=set()"
1
2
3
However, an advantage over | sort | uniq
comes when handling a stream. You see unique lines instantly, without waiting a stream to finish. Useful when using with tail --follow
.
Alternatively, to assure the values are sorted, we can make a use of --end
flag that produces the output after the processing finished.
echo -e "1\n2\n2\n3" | pz "S.add(s)" --end "sorted(S)" -0
Note that we used the variable S
which is initialized by default to an empty set (hence we do not have to use --setup
at all) and the flag -0
to prevent the processing from output (we do not have to use skip
parameter then).
(Strictly speaking we could omit -0
too. If you use the verbose -v
flag, you would see the command changed to s = S.add(s)
internally. And since set.add
produces None
output, it is the same as if it was skipped.)
We can omit (s)
in the main
clause and hence get rid of the quotes all together.
echo -e "1\n2\n2\n3" | pz S.add --end "sorted(S)"
Nevertheless, the most straightforward approach would involve the lines
variable, available when using the --end
clause.
echo -e "1\n2\n2\n3" | pz --end "sorted(set(lines))"
Counting words
We split the line to get the words and put them in S
, a global instance of the set
. Then, we print the set length to get the number of unique words.
echo -e "red green\nblue red green" | pz 'S.update(s.split())' --end 'len(S)' # 3
But what if we want to get the most common words and the count of its usages? Let's use C
, a global instance of the collections.Counter
. We see then the red
is the most_common word and has been used 2 times.
$ echo -e "red green\nblue red green" | pz 'C.update(s.split())' --end C.most_common
red 2
green 2
blue 1
Fetching web content
Accessing internet is easy thanks to the requests
library. Here, we fetch example.com
, grep it for all lines containing "href" and print them out while stripping spaces.
$ echo "http://example.com" | pz 'requests.get(s).content' | grep href | pz s.strip
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
To see how auto-import are resolved, use the verbose mode. (Notice the line Importing requests
.)
$ echo "http://example.com" | pz 'requests.get(s).content' -v | grep href | pz s.strip
Changing the command clause to: s = requests.get(s).content
Importing requests
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
Handling nested quotes
To match every line that has a quoted expressions and print out the quoted contents, you may serve yourself of Python triple quotes. In the example below, an apostrophe is used to delimit the COMMAND
flag. If we used an apostrophe in the text, we would have to slash it. Instead, triple quotes might improve readability.
echo -e 'hello "world".' | pz 'match(r"""[^"]*"(.*)".""", s)' # world
In that case, even better is to use the --match
flag to get rid of the quoting as much as possible.
echo -e 'hello "world".' | pz --match '[^"]*"(.*)"' # world
Computing factorial
Take a look at multiple ways. The simplest is to use the function.
echo 5 | pz factorial # 120
What happens in the background? factorial
is available from math.factorial
. Since it is a callable, we try to put current line as the parameter: factorial(s)
. Since s = "5"
which means a string, it fails. It then tries to use factorial(n)
where n
is current line automatically fetched to a number. That works.
Harder way? Let's use math.prod
then.
echo 5 | pz 'prod(i for i in range(1,n+1))' # 120
Without any built-in library? Let's just use a for-cycle. Process all numbers from 1 to n
(which is 5) and multiply to product. Finally, assign n
to s
which is output.
echo 5 | pz 'for c in range(1,n): n*= c ; s = n' # 120
Using generator will print a factorial for every number from 1 to -g
.
$ pz factorial -g5
1
2
6
24
120
Read CSV
As csv
is one of the auto-imported libraries, we may directly access instantiate the reader object. In the following example, we output the second element of every line either progressively or at once when processing finished.
# output line by line
echo '"a","b1,b2,b3","c"' | pz "(x[1] for x in csv.reader([s]))" # "b1,b2,b3"
# output at the end
echo '"a","b1,b2,b3","c"' | pz --end "(x[1] for x in csv.reader(lines))" # "b1,b2,b3"
Generate random number
First, take a look how to stream random numbers to 100 in Bash.
while :; do echo $((1+$RANDOM%100)); done
Now examine pure Python solution, without having pz
involved.
python3 -c "while True: from random import randint; print(randint(1,100))"
Using pz
, we relieve the cycle handling and importing burden from the command.
pz "randint(1,100)" --generate=0
Let's generate few random strings of variable length 1 to 30. When generator flag is used without a number, it cycles five times.
pz "''.join(random.choice(string.ascii_letters) for _ in range(randint(1,30)))" -S "import string" -g
Average a stream value
Let's have a stream and output the average value.
# print out current line `count` and current average `sum/count`
$ while :; do echo $((1 + $RANDOM % 100)) ; sleep 0.1; done | pz 'sum+=n;s=count, sum/count' --setup "sum=0"
1 38.0
2 67.0
3 62.0
4 49.75
# print out every 10 000 lines
# (thanks to `not i % 10000` expression)
$ while :; do echo $((1 + $RANDOM % 100)) ; done | pz 'sum+=n;s=sum/count; s = (count,s) if not count % 10000 else ""' --setup "sum=0"
10000 50.9058
20000 50.7344
30000 50.693466666666666
40000 50.5904
How can this be simplified? Let's use an infinite generator -g0
. As we know, n
is given current line number by the generator and i
is by default implicitly declared to i=0
so we use it to hold the sum. No setup clause needed. No Bash cycle needed.
$ pz "i+=randint(1,100); s = (n,i/n) if not n % 10000 else ''" -g0
10000 49.9488
20000 50.5399
30000 50.39906666666667
40000 50.494425
Multiline statements
Should you need to evaluate a short multiline statement, use standard multiline statements, supported by Bash.
$ echo -e "1\n2\n3" | pz "if n > 2:
s = 'bigger'
else:
s = 'smaller'
"
smaller
bigger
bigger
Simple progress bar
Simulate a lengthy processing by generating a long sequence of numbers (as they are not needed, we throw them away by 1>/dev/null
).
On every 100th line, we move cursor up (\033[1A
), clear line (\033[K
) and print to STDERR
current status.
$ seq 1 100000 | pz 's = f"\033[1A\033[K ... {count} ..." if count % 100 == 0 else None ' --stderr 1>/dev/null
... 100 ... # replaced by ... 200 ...
Docs
Scope variables
In the script scope, you have access to the following variables:
s
– current line
Change it according to your needs
echo 5 | pz 's += "4"' # 54
n
– current line converted to an int
(or float
) if possible
echo 5 | pz n+2 # 7
echo 5.2 | pz n+2 # 7.2
b
– current line as a byte-string
Sometimes the input cannot be converted to str easily. A warning is output, however, you can still operate with raw bytes.
echo -e '\x80 invalid line' | pz s
Cannot parse line correctly: b'\x80 invalid line'
� invalid line
# use the `--quiet` flag to suppress the warning, then decode the bytes
echo -e '\x80 invalid line' | pz 'b.decode("cp1250")' --quiet
€ invalid line
count
– current line number
# display every 1_000nth line
$ pz -g0 n*3 | pz "n if not count % 1000 else None"
3000
6000
9000
# the same, using the `--filter` flag
$ pz -g0 n*3 | pz -F "not count % 1000"
text
– whole text, all lines together
Not available with the --overflow-safe
flag set nor in the main
clause unless the --whole
flag set.
Ex: get character count (an alternative to | wc -c
).
echo -e "hello\nworld" | pz --end 'len(text)' # 11
When used in the main
clause, an error appears.
$ echo -e "1\n2\n3" | pz 'len(text)'
Did not you forget to use --text?
Exception: <class 'NameError'> name 'text' is not defined on line: 1
Appending --whole
helps, but the result is processed for every line.
$ echo -e "1\n2\n3" | pz 'len(text)' -w
5
5
5
Appending -1
makes sure the statement gets computed only once.
$ echo -e "1\n2\n3" | pz 'len(text)' -w1
5
lines
– list of lines so far processed
Not available with the --overflow-safe
flag set.
Ex: returning the last line
echo -e "hello\nworld" | pz --end lines[-1] # "world"
numbers
– list of numbers so far processed
Not available with the --overflow-safe
flag set.
Ex: show current average of the stream. More specifically, we output tuples: line count, current line, average
.
$ echo -e "20\n40\n25\n28" | pz 's = count, s, sum(numbers)/count'
1 20 20.0
2 40 30.0
3 25 28.333333333333332
4 28 28.25
skip
line
If set to True
, current line will not be output. If set to False
when using the -0
flag, the line will be output regardless.
i
, S
, L
, D
, C
– other global variables
Some variables are initialized and ready to be used globally. They are common for all the lines.
i = 0
S = set()
L = list()
D = dict()
C = Counter()
It is true that using uppercase is not conforming the naming convention. However, in these tiny scripts the readability is the chief principle, every character counts.
Using a set S
. In the example, we add every line to the set and end print it out in a sorted manner.
$ echo -e "2\n1\n2\n3\n1" | pz "S.add(s)" --end "sorted(S)"
1
2
3
Using a list L
. Append lines that contains a number bigger than one and finally, print their count. As only the final count matters, suppress the line output with the flag -0
.
$ echo -e "2\n1\n2\n3\n1" | pz "if n > 1: L.append(s)" --end "len(L)" -0
3
Auto-import
- You can always import libraries you need manually. (Put
import
statement into the command.) - Some libraries are ready to be used:
re.* (match, search, findall), math.* (sqrt,...), defaultdict
- Some others are auto-imported whenever its use has been detected. In such case, the line is reprocessed.
- Functions:
b64decode, b64encode, datetime, (requests).get, glob, iglob, Path, randint, sleep, time, ZipFile
- Modules:
base64, collections, csv, humanize, itertools, jsonpickle, pathlib, random, requests, time, webbrowser, zipfile
- Functions:
Caveat: When accessed first time, the auto-import makes the row reprocessed. It may influence your global variables. Use verbose output to see if something has been auto-imported.
$ echo -e "hey\nbuddy" | pz 'a+=1; sleep(1); b+=1; s = a,b ' --setup "a=0;b=0;" -v
Importing sleep from time
2 1
3 2
As seen, a
was incremented 3× times and b
on twice because we had to process the first line twice in order to auto-import sleep. In the first run, the processing raised an exception because sleep
was not known. To prevent that, explicitly appending from time import sleep
to the --setup
flag would do.
Output
-
Explicit assignment: By default, we output the
s
.echo "5" | pz 's = len(s)' # 1
-
Single expression: If not set explicitly, we assign the expression to
s
automatically.echo "5" | pz 'len(s)' # 1 (command internally changed to `s = len(s)`)
-
Tuple, generator: If
s
ends up as a tuple, its get joined by tabs.$ echo "5" | pz 's, len(s)' 5 1
Consider piping two lines 'hey' and 'buddy'. We return three elements, original text, reversed text and its length.
$ echo -e "hey\nbuddy" | pz 's,s[::-1],len(s)' hey yeh 3 buddy yddub 5
-
List: When
s
ends up as a list, its elements are printed to independent lines.$ echo "5" | pz '[s, len(s)]' 5 1
-
Regular match: All groups are treated as a tuple. If no group used, we print the entire matched string.
# no group → print entire matched string echo "hello world" | pz 'search(r"\s.*", s)' # " world" # single matched group echo "hello world" | pz 'search(r"\s(.*)", s)' # "world" # matched groups treated as tuple echo "hello world" | pz 'search(r"(.*)\s(.*)", s)' # "hello world"
-
Callable: It gets called. Very useful when handling simple function – without the need of explicitly putting parenthesis to call the function, we can omit quoting in Bash (expression
s.lower()
would have had to be quoted.) Use the verbose flag-v
to inspect the internal change of the command.# internally changed to `s = s.lower()` echo "HEllO" | pz s.lower # "hello" # internally changed to `s = len(s)` echo "HEllO" | pz len # "5" # internally changed to `s = base64.b64encode(s.encode('utf-8'))` echo "HEllO" | pz b64encode # "SEVsbE8=" # internally changed to `s = math.sqrt(n)` # and then to `s = round(n)` echo "25" | pz sqrt | pz round # "5" # internally changed to `s = sum(numbers)` echo -e "1\n2\n3\n4" | pz sum 1 3 6 10 # internally changed to `' - '.join(lines)` echo -e "1\n2\n3\n4" | pz --end "' - '.join" 1 - 2 - 3 - 4
As you see in the examples, if
TypeError
raised, we try to reprocess the row while adding current line as the argument:- either its basic form
s
- the
numbers
if available - using its numeral representation
n
if available - encoded to bytes
s.encode('utf-8')
In the
--end
clause, we try furthermore thelines
. - either its basic form
CLI flags
-v
,--verbose
: See what happens under the hood. Show automatic imports and internal command modification (attempts to make it callable and prependings =
if omitted).$ echo -e "hello" | pz 'invalid command' Exception: <class 'SyntaxError'> invalid syntax (<string>, line 1) on line: hello $ echo -e "hello" | pz 'sleep(1)' --verbose Importing sleep from time
-q
,--quiet
: See errors and values only. Suppress command exceptions.echo -e "hello" | pz 'invalid command' --quiet # empty result
Command clauses
COMMAND
: Themain
clause, any Python script executed on every line (multiple statements allowed)-S COMMAND
,--setup COMMAND
: Any Python script, executed before processing. Useful for variable initializing. Ex: prepend line numbers by incrementing a variablecount
.$ echo -e "row\nanother row" | pz 'count+=1;s = f"{count}: {s}"' --setup 'count=0' 1: row 2: another row # the same using globally available variable `count` instead of using `--setup` and the `--format` flag $ echo -e "row\nanother row" | pz -f '{count}: {s}'
-E COMMAND
,--end COMMAND
: Any Python script, executed after processing. Useful for the final output. The variabletext
is available by default here.$ echo -e "1\n2\n3\n4" | pz --end sum 10 $ echo -e "1\n2\n3\n4" | pz s --end sum 1 2 3 4 10 $ echo -e "1\n2\n3\n4" | pz sum --end sum 1 3 6 10 10
-F
,--filter
: Line is piped out unchanged, however only if evaluated toTrue
. When piping in numbers to 5, we pass only such bigger than 3.The statement is equivalent to using$ echo -e "1\n2\n3\n4\n5" | pz "n > 3" --filter 4 5
skip
(and not using--filter
).When not using filter,$ echo -e "1\n2\n3\n4\n5" | pz "skip = not n > 3" 4 5
s
evaluates toTrue
/False
. By default,False
or empty values are not output.$ echo -e "1\n2\n3\n4\n5" | pz "n > 3" True True
-f
,--format
: Main and end clauses are considered f-strings. The clause is inserted in between three-apostrophesf'''COMMAND'''
internally.
Input / output
-
-n NUM
Process only such number of lines. Roughly equivalent tohead -n
. -
-1
Process just the first line. -
-0
Skip all lines output. (Useful in combination with--end
.) -
--empty
Output even empty lines. (By default skipped.)
Consider shortening the text by 3 last letters. First linehey
disappears completely then.$ echo -e "hey\nbuddy" | pz 's[:-3]' bu
Should we insist on displaying, we see an empty line now.
$ echo -e "hey\nbuddy" | pz 's[:-3]' --empty bu
-
-g [NUM]
,--generate [NUM]
Generate lines while ignoring the input pipe. Line will correspond to the iteration cycle count (unless having the--overflow-safe
flag on while having an infinite generator – in that case, lines will equal to '1'). IfNUM
not specified, 5 lines will be produced by default. PuttingNUM == 0
means an infinite generator. If nomain
clause set, the number is piped out.$ pz -g2 1 2 $ pz 'i=i+5' -g -v Changing the main clause to: s = i=i+5 Generating s = 1 .. 5 5 10 15 20 25
-
--stderr
Print clauses' output to theSTDERR
, while letting the original line piped to theSTDOUT
intact. Useful for generating reports during a long operation. Take a look at the following example, every third line will makeSTDERR
to receive a message.$ pz -g=9 s | pz "s = 'Processed next few lines' if count % 3 == 0 else None" --stderr 1 2 3 Processed next few lines 4 5 6 Processed next few lines 7 8 9 Processed next few lines
Demonstrate different pipes by writing
STDOUT
to a file and leavingSTDERR
in the terminal.$ pz -g=9 s | pz "s = 'Processed next few lines' if count % 3 == 0 else None" --stderr > /tmp/example Processed next few lines Processed next few lines Processed next few lines cat /tmp/example 1 2 3 ...
-
--overflow-safe
Preventlines
,numbers
,text
variables to be available. Useful when handling an infinite input.# prevent `text` to be populated by default echo -e "1\n2\n2\n3" | pz --end "len(text)" --overflow-safe Did you not forget to use --while to access `text`? Exception: <class 'NameError'> name 'text' is not defined in the --end clause # force to populate `text` echo -e "1\n2\n2\n3" | pz --end "len(text)" --overflow-safe --whole 7
Regular expressions shortcuts
-
--search
Equivalent tosearch(COMMAND, s)
$ echo -e "hello world\nanother words" | pz --search ".*\s" hello another
-
--match
Equivalent tomatch(COMMAND, s)
-
--findall
Equivalent tofindall(COMMAND, s)
-
--sub SUBSTITUTION
Equivalent tosub(COMMAND, SUBSTITUTION, s)
$ echo -e "hello world\nanother words" | pz ".*\s" --sub ":" :world :words
Using groups
$ echo -e "hello world\nanother words" | pz "(.*)\s" --sub "\1" helloworld anotherwords
Bash completion
- Run:
apt-get install bash-completion jq
- Copy: extra/pz-autocompletion.bash to
/etc/bash_completion.d/
- Restart terminal