Converting Command Line Output to JSON with Nushell

Nushell

Nushell is pretty great.

It’s a new shell that has as its biggest selling point the fact that pipes are inherently structured.

I’ve been playing around with version 0.13.0 and was able to find a few rough edges. But I really will be using this in the future.

Often, I am torn between writing scripts in Bash or Python. Python is obviously much saner for bigger tasks. But Bash operates nicely with the POSIX world and pipes are just great. Nushell looks like a great replacement of Bash (or equivalent) where data is not just raw lines of text.

Converting Output

Last week, I learned off sshping. It’s a small utility that measures SSH statistics (latency, bandwidth). However, its output looks like this:

❯ sshping -H  chris@100.68.54.35
ssh-Login-Time:               1.77  s
Minimum-Latency:              3.71 ms
Median-Latency:               5.58 ms
Average-Latency:              6.03 ms
Average-Deviation:            1.65 ms
Maximum-Latency:              34.6 ms
Echo-Count:                  1.00 kB
Upload-Size:                 8.00 MB
Upload-Rate:                 4.48 MB/s
Download-Size:               8.00 MB
Download-Rate:               4.23 MB/s

In other words, this is a manually constructed table. Not really useful if you want to store it for future analysis.

Nushell is made for handling structured data. But it can also create structured data.

Let me show you how I parse the sshping output to JSON. I will build this up, piece by piece. To speed up the individual steps, I first save the result of sshping into a file. This is not necessary if we are only running the final command once. In case you are wondering, the IP is a NUC machine connected via Tailscale.

sshping -H chris@100.68.54.35 | save --raw ping.raw

The save command is the equivalent of redirecting into a file in Bash (or other POSIX shells). In the same way, there is also open. Note that this is just raw data and note that there are no rows or columns yet.

> open ping.raw
ssh-Login-Time:               1.81  s
Minimum-Latency:              3.14 ms
Median-Latency:               5.82 ms
Average-Latency:              9.26 ms
Average-Deviation:            21.0 ms
Maximum-Latency:               210 ms
Echo-Count:                  1.00 kB
Upload-Size:                 8.00 MB
Upload-Rate:                 3.22 MB/s
Download-Size:               8.00 MB
Download-Rate:               4.82 MB/s

lines will turn each raw line into one row in our data. Each row is numbered:

> open ping.raw | lines
────┬────────────────────────────────────────
 #  │ <value>
────┼────────────────────────────────────────
  0 │ ssh-Login-Time:               1.81  s
  1 │ Minimum-Latency:              3.14 ms
  2 │ Median-Latency:               5.82 ms
  3 │ Average-Latency:              9.26 ms
  4 │ Average-Deviation:            21.0 ms
  5 │ Maximum-Latency:               210 ms
  6 │ Echo-Count:                  1.00 kB
  7 │ Upload-Size:                 8.00 MB
  8 │ Upload-Rate:                 3.22 MB/s
  9 │ Download-Size:               8.00 MB
 10 │ Download-Rate:               4.82 MB/s
────┴────────────────────────────────────────

Next, we turn the rows into columns with the split-column command. This could also be done with the parse command like this: parse {metric}: {value}. Both will give us the same result in this case.

> open ping.raw | lines | split-column ':' metric value
────┬───────────────────┬────────────────────────────
 #  │ metric            │ value
────┼───────────────────┼────────────────────────────
  0 │ ssh-Login-Time    │                1.81  s
  1 │ Minimum-Latency   │               3.14 ms
  2 │ Median-Latency    │                5.82 ms
  3 │ Average-Latency   │               9.26 ms
  4 │ Average-Deviation │             21.0 ms
  5 │ Maximum-Latency   │                210 ms
  6 │ Echo-Count        │                   1.00 kB
  7 │ Upload-Size       │                  8.00 MB
  8 │ Upload-Rate       │                  3.22 MB/s
  9 │ Download-Size     │                8.00 MB
 10 │ Download-Rate     │                4.82 MB/s
────┴───────────────────┴────────────────────────────

Note that the value column has a lot of whitespace that we definitely do not want. find-replace in the str command will help. (NOTE: In the future trim will probably work on cells and be better suited for this job. Version 0.13.0 does not have the fix yet.)

NOTE: See the Update below, str value --trim will only be available in Nu 0.14.0. Replace --trim with --find-replace [\W+ ''] as long as 0.14.0 has not been released.

> open ping.raw | lines | split-column ':' metric value | str value --trim
────┬───────────────────┬───────────
 #  │ metric            │ value
────┼───────────────────┼───────────
  0 │ ssh-Login-Time    │ 1.81  s
  1 │ Minimum-Latency   │ 3.14 ms
  2 │ Median-Latency    │ 5.82 ms
  3 │ Average-Latency   │ 9.26 ms
  4 │ Average-Deviation │ 21.0 ms
  5 │ Maximum-Latency   │ 210 ms
  6 │ Echo-Count        │ 1.00 kB
  7 │ Upload-Size       │ 8.00 MB
  8 │ Upload-Rate       │ 3.22 MB/s
  9 │ Download-Size     │ 8.00 MB
 10 │ Download-Rate     │ 4.82 MB/s
────┴───────────────────┴───────────

And now we can use the save command to save this into the format of our choosing. The file extension determines the format.

> open ping.raw | lines | split-column ':' metric value | str value --trim | save ping.json

Wishlist

Update [2020-04-22]

A str --trim option was added a few hours after I published this by Andrés Robalino. Before the --trim, I had to use --find-replace [\W+ '']

Follow-up

Please see my follow up post Nushell vs Bash!

References

#Nushell