What's the simplest way to _iterate_ over a JSON array using `for` or `while`?

[arielelkin]

What's the simplest way to iterate over a JSON array using for or while?

The documentation doesn't provide simple examples of such iteration using jq

Given this simple JSON:

{
    "users": [
        {
            "id": 1,
            "description": "alfred one",
            "admin": true,
            "height": 10
        },
        {
            "id": 2,
            "description": "alice two",
            "admin": true,
            "height": 11
        },
        {
            "id": 3,
            "description": "emma three",
            "admin": false,
            "height": 33
        }
    ]
}

How can I perform an operation over each object in the array?

For example, just concatenate strings thus:

alfred one is 10
alice two is 11
emma three is 33

I tried this:

for i in "$(jq -r '.users[]' users.json)"; do
    name=$(echo $i | jq -r .description)
    height=$(echo $i | jq -r .height)
    echo "$name has id $height" 
done

But this produces this:

alfred one
alice two
emma three is 10
11
33

(I may be able to use join in this specific instance, but that's besides the point, I am asking for a generic solution to use jq in while and for loops)

Also consider updating the documentation accordingly, this has caused some confusion in the past (see https://github.com/stedolan/jq/issues/2169 and https://github.com/stedolan/jq/issues/1137)

[itchyny]

You can use NUL-separated output.

while IFS= read -r -d $'\0' user_json; do
  name=$(echo "$user_json" | jq -r .description)
  height=$(echo "$user_json" | jq -r .height)
  echo "$name has id $height" 
done < <(jq --nul-output '.users[]' users.json)

[arielelkin]

@itchyny sorry but that produces the following error:

produce_variants.sh: line 24: syntax error near unexpected token `<'
produce_variants.sh: line 24: `done < <(jq --nul-output '.users[]' users.json)'

[itchyny]

Oops, I assumed Bash.

[itchyny]

How about

jq --nul-output '.users[]' users.json | \
  while IFS= read -r -d $'\0' user_json; do
    name=$(echo "$user_json" | jq -r .description)
    height=$(echo "$user_json" | jq -r .height)
    echo "$name has id $height" 
  done

[lfmwuy]

#!/bin/bash
Users_json=`jq '.' users.json`
Users_length_end=`jq -r ".users[]| length" users.json|wc -l`
users_length_start=0
while (( $users_length_start < $Users_length_end ))
do
        id=$(echo $Users_json | jq -r ".users[$users_length_start].id")
        echo "id=$id"
        users_length_start=`expr $users_length_start + 1`
done

[arielelkin]

@itchyny your code produces the following error:

jq: Unknown option --nul-output

@lfmwuy's code works, but unfortunately it's too convoluted for an operation that simple. However it gave me an idea and I found a simpler way:

count=`jq '.users | length' users.json`

for ((i=0; i<$count; i++)); do
    name=`jq -r '.users['$i'].description' users.json`
    height=`jq -r '.users['$i'].height' users.json`
    echo "$name is $height"
done

Please consider closing this issue after adding this snippet to the manual! This has caused confusion in the past (see #2169 and #1137)

[pkoppstein]

Some of the proposed solutions in this thread are unnecessarily and even grossly inefficient, but the topic of iterations as it relates to jq and various shells is too large for a short summary here, so I would just like to point out that:

(1) often whatever iteration is required can be accomplished in jq without shell variables or shell iterative constructs;

(2) when it is necessary to use a shell iterative construct, it is usually possible to avoid calling jq more than once.

(3) There are already many excellent online resources covering these and related topics, e.g. on stackoverflow.com:

https://stackoverflow.com/questions/48005870/iterate-over-json-with-jq https://stackoverflow.com/questions/33950596/iterating-through-json-array-in-shell-script https://stackoverflow.com/questions/54952736/how-to-iterate-a-json-array-in-bash-from-jq

Also, please consider that although ERs are certainly welcome here, stackoverflow.com is usually a more suitable forum for usage questions.

[pkoppstein]

@arielelkin - Regarding your comment about --nul-output:

(1) This is a documented feature of the "development" version of jq (https://stedolan.github.io/jq/manual/)

(2) There are several versions of jq, and each online edition of the official "jq manual" contains links to other editions at the very top, e.g. the "development" version of the jq manual begins as follows:

For released versions, see jq 1.6, jq 1.5, jq 1.4 or jq 1.3.

[arielelkin]

@pkoppstein

The existence of at least three stackoverflow questions asking how to carry out such a simple task, with more than 50,000 combined views, should highlight the glaring gap in jq's documentation. Very basic stackoverflow questions about a library are not a replacement for its documentation, it just shows the library is under-documented.

the topic of iterations as it relates to jq and various shells is too large for a short summary here

The point of a library like jq is to abstract away this complexity. Iterating over an array of json objects is one the most rudimentary tasks any user would expect out of a library like jq. If jq can't make simple tasks simple, then consider revising its API. Or updating its documentation.

[pkoppstein]

This response to the original question focuses on:

1. a jq-only solution

2. a jq+bash solution

1. jq-only

    jq -cr '.users[] | "\(.description) is \(.height)"' sample.json

Here, the larger point is that .[] can serve as an iterator because of jq's backtracking semantics. range and inputs are other frequently used generators. For reduction, reduce is usually sufficient. The jq manual describes these and other building blocks (e.g. foreach, while, and until) for iteration within jq.

2. jq+bash

while read -r description
do
    read -r height
    echo "$description is $height"
done < <( jq -cr '.users[] | (.description, .height)' sample.json)

Here the larger point is that there is often no need to invoke jq more than once. If the strings contain embedded newlines, then this solution could be easily adapted to use "\u0000" as the separator character.

[dreusel]

@arielelkin your question has very little to do with JQ itself and is more about 'how do I use my shell'. The fact that you're not able to distinguish between the two makes it very clear that you're in no position to criticize authors of tools such as JQ, that are used by pretty much anyone who knows (and doesn't know) what they're doing.

Stack overflow is full of similar questions from people who can't figure out how to use a combination of tools for their specific use-case. It doesn't mean that all of those tools are 'under-documented'. It just means that A) they are very popular and B) by spelling out all those solutions like that we've created a world where it's become normal to expect copy-pastable solutions for any problem instead of properly learning the tools we're using and figuring out things for ourselves. A world where people who lack essential skills such as generalizing concepts manage to produce some working programs and therefore start thinking of themselves as 'programmers'. Of course the next step was going to be those 'programmers' blaming the authors for not providing a copy-pastable solution for every situation they encounter.

Managing to copy-paste together working programs does not make you a programmer. It does not qualify you for talking to programmers as if you know what you're talking about. Stop complaining and start learning the skills essential to your job, or find another one.

[arielelkin]

Normally, I'd dismiss the kind of bile just spouted by @dreusel, especially in the context of a github issue. However, underlying his personal attacks, self-satisfaction, and bitterness, I recognise some notions and attitudes that aren't just wrong, they're also positively harmful to the industry. Newcomers to the field, or anyone else trying to form an opinion on these matters, must not infer that this epistemic abuse is the norm, so it'd be irresponsible of me, with 10 years' experience as a professional software engineer, to remain silent in the face of it.

@dreusel You are fundamentally wrong about the merits of copy-pasting code. One cannot possibly remember the syntax and semantics of the hundreds of APIs (the shell's, the language's, the OS's, the SDKs, the remote APIs, etc.) used on a daily basis. And even if one could, those APIs will still produce runtime or buildtime issues that are quicker to fix by researching solutions from Stackoverflow than by manual debugging. Copy-pasting is not just necessary standard practice, it's also a virtuous one.

As evidence, see here thousands of seasoned programmers agreeing: https://twitter.com/i/events/835969005668220930

Here is one of the most eloquent tweets:

"Computer Science is half-remembering something and googling the rest"

Indeed, the reason for which computer science has made such massive progress in such little time is in no small part due to the fact that programmers could copy-paste code (either as snippets or as entire libraries) to integrate them into new solutions. No other discipline has a quicker native way of transferring knowledge. The practice of copying and pasting code is actually one of the pillars of software engineering practice.

The notion that copy-pasting implies a "lack of essential skills" or an inability to "figure things out for ourselves" is not just wrong, it's also disdainful towards your peers. I'll grant that some instances of copy-pasting may indeed be symptomatic of a programmer's laziness, but your generalised scorn for the practice shows the kind of condescending attitude towards your fellow programmer that has been plaguing our industry.

You sound like the kind of "brogrammer" who's badly affected by the curse of knowledge (it's a thing, look it up). Complacently writes superclever code and doesn't comment it because it's clear enough in his head, therefore it ought to be clear enough in the next programmer's head. When you leave the company you invariably leave behind a hair-pulling codebase that's mostly incomprehensible to whichever poor soul inherits it. Principles like clarity at the callsite, KISS, or local reasoning obviously don't mean much to you – people need to "properly learn the tools". You're wrong.

An API is a contract between machines whereby a certain command must result in a certain output, and also between humans whereby I as your user agree to become dependent on your work. I as a user deposit trust in an API vendor's competence; I expect the API to be dependable. Which implies, you guessed it, well documented.

I think you are somewhat aware of this.. A few weeks ago you created an issue and submitted a PR in another repo asking for... clearer documentation. Imagine if that was met with a rabid and condescending tirade telling you to "stop complaining".

Your GitHub profile indicates you're a Tech Lead at a company that makes an ~IFTTT clone~ developer-facing product. Well, I pity the devs using it (and your minions too). But hey hopefully for your users, your API will become "very popular" so that Stackoverflow will take over your job of documenting it properly. Because you're clearly too busy scouring stale GitHub issues on repos you never contribute code to, telling people there that there's a difference between a shell and a shell command so RTFM and GTFO. If there's one thing your users will "figure out for themselves", it's that your competitor's API is most certainly better designed and documented than yours. I highly doubt, for instance, that your API docs also feature a "Was this page helpful" set of buttons on every page.

Beware the curse of knowledge. The criteria and metrics for what makes good documentation don't come from an API's vendor, they actually come from an API's users. It doesn't matter what you as an API vendor think about the quality of your documentation. If your users say your documentation is lacking, it means your documentation is lacking and must be improved, period.

I've worked on three separate developer-facing SDKs for three different companies, I've contributed to the documentation of a handful of big and small open source projects. I am a maintainer of two open-source projects with more than 2k GitHub stars combined, installed in apps with userbases of millions. I can tell you that those stars and that trust were not earned by telling my users to RTFM and reconsider their career choices.

Every one of my users' requests for a documentation improvement is not just legitimate, it's a blessing for my API. It's my users showing me an easy way to make my API more usable. It involves a pretty small investment of time from my part as a vendor, but the returns are huge.

Because this is a world where people transfer knowledge via copy-pasting, where basic functionality of an API (like how to use it within a freaking for loop) ought to be documented, and where users of an API are entitled to demand better documentation. And if you don't like it, then this world is definitely not for you and I hope that you, my dear @dreusel, go contaminate another one.

[vintnes]

@pkoppstein jq -cr '.users[] | "(.description) is (.height)"' sample.json

Pardon me if I've missed something but I think the string interpolation backslashes may have been lost in translation and --compact-output is unnecessary with string outputs, though it certainly doesn't hurt anything.

jq -r '.users[] | "\(.description) is \(.height)"' sample.json

@arielelkin Didn't read your novel. One of the principle advantages of JQ is how iteration is, in fact, so simple and so fundamental to the way that expressions are constructed that it completely obviates the idea of a "for loop". foreach is an advanced construct in this language. .[] is documented under Basic Filters.

[giovanni-nabil]

From this answer provided by felipecrs If you're using Bash, you can do something like this

readarray -t users < <(jq -c '.users[]' users.json) #convert to bash arrays and remove trailing newline for each line (if any)

IFS=$'\n' #read til newline
for user in ${users[@]}; do
    name=$(jq '.description' <<< "${user}")
    height=$(jq '.height' <<< "${user}")
    echo "$name has id $height"
done
unset IFS

[phgSW61HS]

Normally, I'd dismiss the kind of bile just spouted by @dreusel, especially in the context of a github issue. However, underlying his personal attacks, self-satisfaction, and bitterness, I recognise some notions and attitudes that aren't just wrong, they're also positively harmful to the industry. Newcomers to the field, or anyone else trying to form an opinion on these matters, must not infer that this epistemic abuse is the norm, so it'd be irresponsible of me, with 10 years' experience as a professional software engineer, to remain silent in the face of it.

@dreusel You are fundamentally wrong about the merits of copy-pasting code. One cannot possibly remember the syntax and semantics of the hundreds of APIs (the shell's, the language's, the OS's, the SDKs, the remote APIs, etc.) used on a daily basis. And even if one could, those APIs will still produce runtime or buildtime issues that are quicker to fix by researching solutions from Stackoverflow than by manual debugging. Copy-pasting is not just necessary standard practice, it's also a virtuous one.

As evidence, see here thousands of seasoned programmers agreeing: https://twitter.com/i/events/835969005668220930

Here is one of the most eloquent tweets:

"Computer Science is half-remembering something and googling the rest"

Indeed, the reason for which computer science has made such massive progress in such little time is in no small part due to the fact that programmers could copy-paste code (either as snippets or as entire libraries) to integrate them into new solutions. No other discipline has a quicker native way of transferring knowledge. The practice of copying and pasting code is actually one of the pillars of software engineering practice.

The notion that copy-pasting implies a "lack of essential skills" or an inability to "figure things out for ourselves" is not just wrong, it's also disdainful towards your peers. I'll grant that some instances of copy-pasting may indeed be symptomatic of a programmer's laziness, but your generalised scorn for the practice shows the kind of condescending attitude towards your fellow programmer that has been plaguing our industry.

You sound like the kind of "brogrammer" who's badly affected by the curse of knowledge (it's a thing, look it up). Complacently writes superclever code and doesn't comment it because it's clear enough in his head, therefore it ought to be clear enough in the next programmer's head. When you leave the company you invariably leave behind a hair-pulling codebase that's mostly incomprehensible to whichever poor soul inherits it. Principles like clarity at the callsite, KISS, or local reasoning obviously don't mean much to you – people need to "properly learn the tools". You're wrong.

An API is a contract between machines whereby a certain command must result in a certain output, and also between humans whereby I as your user agree to become dependent on your work. I as a user deposit trust in an API vendor's competence; I expect the API to be dependable. Which implies, you guessed it, well documented.

I think you are somewhat aware of this.. A few weeks ago you created an issue and submitted a PR in another repo asking for... clearer documentation. Imagine if that was met with a rabid and condescending tirade telling you to "stop complaining".

Your GitHub profile indicates you're a Tech Lead at a company that makes an ~IFTTT clone~ developer-facing product. Well, I pity the devs using it (and your minions too). But hey hopefully for your users, your API will become "very popular" so that Stackoverflow will take over your job of documenting it properly. Because you're clearly too busy scouring stale GitHub issues on repos you never contribute code to, telling people there that there's a difference between a shell and a shell command so RTFM and GTFO. If there's one thing your users will "figure out for themselves", it's that your competitor's API is most certainly better designed and documented than yours. I highly doubt, for instance, that your API docs also feature a "Was this page helpful" set of buttons on every page.

Beware the curse of knowledge. The criteria and metrics for what makes good documentation don't come from an API's vendor, they actually come from an API's users. It doesn't matter what you as an API vendor think about the quality of your documentation. If your users say your documentation is lacking, it means your documentation is lacking and must be improved, period.

I've worked on three separate developer-facing SDKs for three different companies, I've contributed to the documentation of a handful of big and small open source projects. I am a maintainer of two open-source projects with more than 2k GitHub stars combined, installed in apps with userbases of millions. I can tell you that those stars and that trust were not earned by telling my users to RTFM and reconsider their career choices.

Every one of my users' requests for a documentation improvement is not just legitimate, it's a blessing for my API. It's my users showing me an easy way to make my API more usable. It involves a pretty small investment of time from my part as a vendor, but the returns are huge.

Because this is a world where people transfer knowledge via copy-pasting, where basic functionality of an API (like how to use it within a freaking for loop) ought to be documented, and where users of an API are entitled to demand better documentation. And if you don't like it, then this world is definitely not for you and I hope that you, my dear @dreusel, go contaminate another one.

@dreusel I agree with some of the poin ts you highlighted, but I would rephrase it: copy/pasting in itself is not an issue but there is big difference between software engineers that clearly figure out the underlying concepts of the snippets in question , and those who don't. Nevertheless, you are definitely condescendent and look far away from nowadays reality when you consider that people should master deeply all the aspects of every libraries or apis they have to use within their projects. Things have clearly changed quickly in the industry: 10-15 years ago computer scientists used to work in well defined scoped (like Java Backend Developer, DBA, system engineer...). As of 2014, after 2 years of Software Engineering in .NET, I started working as a Data/Devops Engineer in Data Lake implementations (First in Hadoop ecosystem for 4,5 years, then in diverse clustered/distributed platform like Kubernetes, GKE, Openshift or GCP Paas...), and the technological scoped just kept growing. On monday you work on building CI/CD pipelines in Jenkis (so coding in Groovy), on Tuesday you create helm charts to deploy Kafka on kubernetes, implenting mutual TLS, on Wednesday you set up filebeat or fluentd agents to collect the JMX metrics of the kafka brokers through jolikia , Thursday you create logstah pipeline to parse the JMX messages and ingest them in Elasticsearch, and then create a dashboard in Kibana, Friday you develop a spark application in Scala that will push datya in Kafka topics, Next monday you develop kafka cliebnts in Java that will consume those topics and persist data in mongoDB, and the next tueasday you develop an api in node.js leveraging the power of asynchronous callbacks using promises, and last but not least you realize that you also need to create an openshift job that will set acls on the topics based on specifications stored in a Json conf file, and to do so you go for a shell script that will use jq ... So definitely you cannot expect people to master every single apis like it zas possible in early 2000's where people used to work in smaller silos. So either you are the smartest guy I have ever seen or you're just a bullshitter that enjoys spitting at people's face

[nicowilliams]

@arielelkin

: ; jq -r '.users[] | "\(.description) has id \(.height)"' users.json
alfred one has id 10
alice two has id 11
emma three has id 33
: ; jq -r '.users[] | "\(.description) has id \(.height)" | @sh' users.json
'alfred one has id 10'
'alice two has id 11'
'emma three has id 33'
: ; n=$(jq '.users|length' users.json); for ((i = 0; i < n; i++)); do eval echo $(jq -r --argjson i $i '.users[$i] | "\(.description) has id \(.height)" | @sh' users.json); done
alfred one has id 10
alice two has id 11
emma three has id 33

It's true that shell scripting around jq can be hard at times. That's one reason why I want to finish the branch that has co-routines and plugins, because that also has sub-process support and one could do more scripting in jq itself. Anyways, for now the @sh thing should help you.

[nicowilliams]

@itchyny, @owenthereal, feel free to timeout or ban users who are rude.

[pkoppstein]

@itchyny - I would like to propose closing this issue for the following reasons.

As per the OP's remark at https://github.com/jqlang/jq/issues/2247#issuecomment-759627989, the issue boils down to a documentation enhancement request, which however falls outside the scope of the jq manual and tutorial: the main topic of the request (using jq with other tools) is very broad and generally falls within the purview of the jq FAQ and Cookbook rather than the jq manual or tutorial.

In addition, contributions can be made to the jq FAQ and/or Cookbook without requiring the existence of an "Issue" in this space.

It may also be germane to point out that:

a) anyone who wishes to propose a specific enhancement to the official jq manual or tutorial can still do so (e.g. via a Pull Request), even after this issue is closed;

b) the addition to the documentation that @arielelkin proposed in the above-mentioned post of Jan 13, 2021 is more of an anti-pattern than a pattern to be emulated, not least because it involves three calls to jq when only one is warranted.