PlannedFor lookup is limited to the first 1000 iterations

[Mershl]

Hi @dixudx,

I'm seeing the following behaviour in Jazz 6.0.3.

With an example server containing 2738 iterations the lookup of the iterations in _getPlannedFors will only return the first 1000 iterations and thus resulting in an error if creating an WorkItem of a newer iteration.

This happens because the first request results in:

<oslc_cm:Collection oslc_cm:next="jazzhost.com/ccm/oslc/iterations?oslc_cm.pageSize=100&_resultToken=_IS2hQHgeEee2bfthPc5aiA&_startIndex=100" oslc_cm:totalCount="2738">

But following the next link the totalCount has changed and thus limiting the logic in client.py:1410 to the first 1000 iterations.

<oslc_cm:Collection oslc_cm:next="jazzhost.com/ccm/oslc/iterations?oslc_cm.pageSize=100&_resultToken=_IS2hQHgeEee2bfthPc5aiA&_startIndex=200" oslc_cm:previous="jazzhost.com/ccm/oslc/iterations?oslc_cm.pageSize=100&_resultToken=_IS2hQHgeEee2bfthPc5aiA&_startIndex=0" oslc_cm:totalCount="1000">

I'm experimenting on how to avoid this limit.

[dixudx]

@Mershl This seems to be an issue of RTC, not our client.

Anyway we should have a workaround. It seems depending on the returned totalCount is fragile.

ISTR, RTC has some limitations on the resources count.

• OSLC REST interface only returns 1000 entries;
• How to reset the maximum workitem query result size in Rational Team Concert;

If so, we have nothing to do except manually resetting the maximum workitem query result size in Rational Team Concert. However there is still a maximum limitation there.

[Mershl]

@dixudx An option would be to only lookup resources that are part of the current requested projectarea.

Do you think it's possible to query for something like:

'rtc_cm:timeline/rtc_cm:projectArea="_qi848CJaEeZy9NmiE0zZ4Q"'

in client.py:1341?

[Mershl]

Passing rtc_cm:timeline/rtc_cm:projectArea="_qi848CJaEeZy9NmiE0zZ4Q" as a parameter works as a Short-Term fix in my case reducing the amount of Iterations to ~700.

Example Request URL: https://jazzhost.com/ccm/oslc/iterations?oslc_cm.pageSize=100&_startIndex=0&oslc_cm.query=rtc_cm%3Atimeline%2Frtc_cm%3AprojectArea%3D%22_qi848CJaEeSy9NmiE0zU4Q%22

Edit: This will cause a HTTP 400 Error if used for every _get_paged_resource requests. But works for iteration and possible other requests.

[dixudx]

@Mershl Looks good. Great.

[Mershl]

@dixudx Unfortunately this does not fix all problems. With Jazz 6.0.3 on a full server the lookup on Task creation is currently showing the following output:

2017-08-03 15:59:35,194 ERROR client.RTCClient: No TeamArea named xxx
2017-08-03 15:59:35,194 ERROR client.RTCClient: No TeamArea named xxx
2017-08-03 15:59:36,457 ERROR client.RTCClient: No Severity named xxx
2017-08-03 15:59:36,457 ERROR client.RTCClient: No Severity named xxx
2017-08-03 15:59:42,108 ERROR client.RTCClient: No FiledAgainst named xxx
2017-08-03 15:59:42,108 ERROR client.RTCClient: No FiledAgainst named xxx
2017-08-03 15:59:43,410 ERROR client.RTCClient: 'RESTClient' object has no attribute 'getSubscribers'
2017-08-03 15:59:43,410 ERROR client.RTCClient: 'RESTClient' object has no attribute 'getTaskType'
2017-08-03 15:59:43,410 ERROR client.RTCClient: 'RESTClient' object has no attribute 'getSubject'

resulting in

requests.exceptions.HTTPError: 409 Client Error: Conflict

[dixudx]

Seems you're querying some unexisted fields.

[dixudx]

@Mershl How come rtc_cm:timeline. I think this is unnecessary.

It's true that adding projectArea in the query will help solve this issue to some extent.

[Mershl]

Hi @dixudx,

I've checked the official Java API Client and they provide so called "huge queries" that allow to get results bigger than 1000 items. Do you have an idea how we can solve this problem?

[dixudx]

@Mershl Any detailed doc on that huge queries?

[Mershl]

@dixudx Unfortunately no, it's just a workaround where the query is adapted (filter is stricter) to limit the number of results and concat them all together - in the end you still have x queries.

[dixudx]

@Mershl Would you mind sharing the examples/workarounds? Thanks.

[Mershl]

@dixudx Unfortunatly I no longer have access to the source code of this workaround. For now I use this workaround to get back to speed on my project.

(We still need to specify the query for PlannedFor including rtc_cm:timeline, but not for FiledAgainst)

        if resource_name != "Query" and projectarea_id is not None and resource_name in ['PlannedFor']:
            # PlannedFor does not allow query by projectArea alone.
            resource_url += '&oslc_cm.query=rtc_cm:timeline/rtc_cm:projectArea=\"{0}\"'.format(projectarea_id)

        if resource_name != "Query" and projectarea_id is not None and resource_name in ['FiledAgainst']:
            resource_url += '&oslc_cm.query=rtc_cm:projectArea=\"{0}\"'.format(projectarea_id)

[dixudx]

@Mershl Seems we can only rely on rtc_cm:projectArea right now.