node-osmosis icon indicating copy to clipboard operation
node-osmosis copied to clipboard

how can return values be guaranteed?

Open tripower opened this issue 2 years ago • 4 comments

is it possible that a return value is guaranteed? because if one "find" doesn not find anything the whole data is empty so I tried this code but it only returns the first value??

const osmosis = require('osmosis');
let savedData = [];

var objSite = osmosis.get('https://www.wallstreet-online.de/etf/a0rpwh-ishares-core-msci-world-ucits-etf');
objSite.find("//*[@class='marketchange'][1]/div[1]/div[1]/div[1]/span")
    .set({'kurs': 'text()'})
    .data(data => {
        console.log("data1:" + data);
        savedData.push(data);
        objSite.find("//*[@class='marketchange'][1]/div[1]/div[3]/span")
                .set({'perc': 'text()'})
                .data(data => {
                      console.log("data2:" + data);
                      savedData.push(data);
                })
			    .done(function() {
				    var strFileData = JSON.stringify( savedData, null, 4);
				    console.log("strFileData:" + strFileData);
				    fs.writeFile('data.json', strFileData, function(err) {
					 if(err) console.error(err);
					 else console.log('Data Saved to data.json file');
				   })
				});
    });

=> output strFileData:[ { "kurs": "71,39" } ]

original example which works if all xpath queries find something

    objSite.find("//*[@class='marketchange'][1]/div[1]/div[1]/div[1]/span")
    .set({'kurs': 'text()'})
	.find("//*[@class='marketchange'][1]/div[1]/div[3]/span")
	.set({'perc': 'text()'})
	.data(data => {
	  console.log("data2:" + data);
	  savedData.push(data);
	})
	.done(function() {
		var strFileData = JSON.stringify( savedData, null, 4);
		console.log("strFileData:" + strFileData);
		fs.writeFile('data.json', strFileData, function(err) {
		 if(err) console.error(err);
		 else console.log('Data Saved to data.json file');
	   })
	});
```	
=> output
strFileData:[
    {
        "kurs": "71,39",
        "perc": "+0,08"
    }
]

tripower avatar Jul 19 '22 16:07 tripower

the first problem is that you use the same objSite object for two separate chains, so you overwrite the previous defined data/done functions (is i understand) -> you should put them together:

objSite
    .find("selector1")
    .set()
    .find("selector2")
    .set()
    .data((dataForBothSelectors) => {})
    .done()

BUT: with this solution if e.g. selector1 was not found in the DOM the chain breaks and done is called (works as implemented, s. lib/commands/find.js:42).

only solution i know would be to create multiple separate objSite objects (multiple site visits), then you can ignore if one does not resolve any value.

or is there a functionality in the library to ignore find errors?

jueschus avatar Jul 20 '22 14:07 jueschus

thx

yes would be a cool festure with a flag that wrong xpath will return empty value instead that the whole data is empty if one fails

tripower avatar Jul 21 '22 08:07 tripower

my local HACK to get a result as desired hack for osmosis

tripower avatar Jul 23 '22 17:07 tripower

you can use "do" for this

only the "do" block will fall if the selector is not found

example:

      osmosis
        .get(url)
        .delay(1000)
        .set({
          text: ['.topic-body .topic-body__content-text'],
          videoId: '.topic-body .box-external-video .box-external-video__eagle@data-vid',
          title: '.topic-body__title'
        })
        .do(
          osmosis
          .follow('.topic-body .topic-body__title-image-zoom@href')  // --> **this selector may be missing. **
          .delay(1000)
          .set({src: '.comments-page__body .comments-page__title-image@src'})
        )
        .data(resolve)
        .error(reject)
        .debug(console.log)
        .done()
    })

Ruinevo avatar Mar 28 '24 01:03 Ruinevo