Error during query with satisfies links

[sergiomarsen]

Hello, I have been using the oslc query for retrieving artifacts attribute from DNG /rm.

When using global configuration, I am executing a query to retrieve artifacts with links of type 'Satisfies'. To do that I execute a query as the following one:

oslcquery -J https://my.server:port -U usr -P pwd -A rm,gc,jts:jts23 -p "req_project_name" -E "gc_project_name" -G "gc_stream" -O data/proj/SWreqs.csv -s "dcterms:identifier,oslc:instanceShape,rm:module,Satisfies" -W --pagesize 0 

With this, I am getting the following error:

Version 0.19.0
querying for gc config **proj** V3.0
Loading DN shapes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 70/70 [00:12<00:00,  5.62 results/s]
Preparing Query
Query completed in 1 page(s)
Processing       :  55%|█████████████████████████████████████████████████████████████████████████████████████████▊                                                                         | 14796/26837 [00:47<07:03, 28.44 results/s]T
raceback (most recent call last):
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\Scripts\oslcquery.exe\__main__.py", line 7, in <module>
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\examples\oslcquery.py", line 808, in main
    do_oslc_query(sys.argv[1:])
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\examples\oslcquery.py", line 569, in do_oslc_query
    results = queryon.do_complex_query( args.resourcetype, querystring=args.query, searchterms=args.searchterms, select=args.select, isnulls=args.null, isnotnulls=args.value
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\oslcqueryapi.py", line 214, in do_complex_query
    remappedvalue.append(self.resolve_uri_to_name(lv))
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\_project.py", line 307, in resolve_uri_to_name
    if ( result := self.app_resolve_uri_to_name(uri) ) is None:
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\_rm.py", line 751, in app_resolve_uri_to_name
    result = self.resource_id_from_uri(uri)
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\_rm.py", line 797, in resource_id_from_uri
    resource_xml = self.execute_get_rdf_xml(reluri=uri, intent="Retrieve type RDF to get its id (dcterms:identifier)")
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\httpops.py", line 152, in execute_get_rdf_xml
    response = request.execute( **kwargs )
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\httpops.py", line 317, in execute
    return self._execute_request( no_error_log=no_error_log, close=close, **kwargs )
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\httpops.py", line 326, in _execute_request
    result = self._execute_one_request_with_login( no_error_log=no_error_log, close=close, **kwargs)
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\elmclient\httpops.py", line 531, in _execute_one_request_with_login
    response.raise_for_status()
  File "C:\Users\sergio.marcos2\.conda\envs\kpienv\lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: **base_url**/rm/resources/BI_Jj_gc4qEEe2X94hUXsF8pw?oslc_config.context=**base_url**%3A9443%2Fgc%2Fconfiguration%
2F71
Processing       :  55%|█████████████████████████████████████████████████████████████████████████████████████████▎                                                                        | 14796/26837 [00:47<00:38, 311.40 results/s]

And actually, after checking, is true that with the given URI(if I check at navigator) there is no artifact with that URI UUID among that global configuration. So checking further, in previous gc streams it was and I detected the id(and the requirement itself) is in the stream I'm looking for, but the URI UUID seems that changed between gc streams. So it is something like this status:

At GC stream 2.0 -> ID: 356124 || UUID: BI_Jj_gc4qEEe2X94hUXsF8pw At GC stream 3.0 -> ID: 356124 || UUID: BI_XBPSkwM_Ee6lBfMIxW_p0w

I do not know, if for this change at the uri UUID it might be problems when quering, I have cheked the link at doors and it is okey, even tried removing link and adding again.

So I'd like to ask if you think this could be the reason why this error is appearing or you would know an approach to solve it, as when the error appears no results gets generated, so I cannot check for other values if are being properly processed(I guess yes), But I used similar queries for other projects and worked pretty fine.

-- Also, I am not querying here by any artifact type because the ones i need to look for, has same name in different components of the project but some of them have URIs defined and others not and I cannot change as some are already baselined (Querying by the artifact type name within global configuration do not return the needed result).

Thanks a lot in advance. Regards!

[barny]

Hi

The UUIDs are different because each binding (use of an artifact in a module) has a different UUID. The ID is the same because the bindings are referring to the same core/base artifact.

There's a limitation of oslcquery that it doesn't know how to recognise the same type in different components (i.e. in a GC). Defining URIs on the type in each component and oslcquery using the URI is how it should work, I think, but I don't have time to implement that ATM. You should anyway be defining URIs because Report Builder needs these to match the types up too.

However the issue you're seeing seems different.

[sergiomarsen]

Thanks for your reply barny,

Yes from now on we are defining URIs for every element, but we cannot do it for some already baselined streams.

Anyway, I finally I did a workaround by doing one oslcquery for every component (with local configurations) and then combined them in my side. With that I were able to take the satisfies links with the uuids and relate them in this way.

For now I'm manually taking the local config stream/baseline uri of every module which take part in a global stream and using them for all the extractions I need. In this way I avoid any errors which gc usage may suppose but I cannot ensure that the local stream/baseline is still in that gc.

So my idea now is to automatically extract all the streams/baselines that are inside a gc stream to avoid introducing this data manually at some constants (as this changes during the project development). Probably it can be done somehow with oslcquery or with some simple API requests but I still need to work on it.

Thank you for your advice!

[barny]

That sounds like a pragmatic approach :-)

FYI GCM has documented APIs on https://YOURSERVER:PORT/gc/doc/scenarios

There's an API call to get the contributions to a GC hierarchy see https://jazz.net/gc/doc/scenario?id=GetFlatListOfContributionsForGcHiearchy

[barny]

There's existing code in oslcquery.py which is an example of getting and processing all the RM contributions to a gc - look for percontribution - The reason this code was added was to use threading to do the queries in parallel which made a significant speedup for a gc with many large RM contributions. You can enable the threading by using option --threading

You might want to try the --percontribution option because instead of making a single OSLC query on the GC this makes an OSLC query on each contribution to the GC using the local config - so types will be resolved for each local config. If the contributions aren't all in the same project use --crossproject to bring them into scope.

[sergiomarsen]

Thank you so much for this hint! :)

That would really do what I was trying to figure out in a far more simple way.

I will have a look on it when I have more time but it really seems the best approach as I've just read it's also extracting the module in a column which is something we need too. I guess the best way would be using -s * to avoid problems among contributions. And --crossproject will be also needed for some projects so thanks for noting this too.

Also threading is something which was in our plans so I'll test it too and if everything adapts fine I will use it for our project.

Thank you!

---

Also one note, for one project, I needed to increase a range at rdfxml.py for prefixes, as for some of ours qm extractions it was not enough:

It's probably a miss-configuration from our side, but I used this little change as a workaround, just in case is useful for you to know :)

[barny]

Thanks - next release will remove the hard limit 1000, will simply iterate incrementing until it finds a unique new prefix.