WikibaseImport

WikibaseImport copied to clipboard

Hi guys,

First of all, thanks for useful tool. It helps me a lot.

Today, I can't import some entity with metre (Q11573), square kilometre (Q712226), such as Q1854

It's ok with entity Q36404

$ php maintenance/importEntities.php --entity Q36404
...
[2016-12-22 10:05:39]: Adding statements: Q36404
[2016-12-22 10:05:44]: Done

But it yield Illegal value: http://www.wikidata.org/entity/Q11573 when I try to dump entity Q1854

$ php maintenance/importEntities.php --entity Q1854

...
[2016-12-22 10:08:57]: Adding statements: Q1854
[2016-12-22 10:09:00]: Illegal value: http://www.wikidata.org/entity/Q11573
[2016-12-22 10:09:00]: Done

Entity Q11573 is metre, I realize that I can't dump any entities with metre (Q11573), square kilometre (Q712226). For example

$ php maintenance/importEntities.php --entity Q36685

...
[2016-12-22 10:12:52]: Adding statements: Q36685
[2016-12-22 10:12:55]: Illegal value: http://www.wikidata.org/entity/Q712226
[2016-12-22 10:12:55]: Done

I walk through this issue by ignore some units

At addStatementList function in StatementsImporter.php, add these lines

$items = array();
foreach($data as $item){
    $is_valid = true;
    try {
        $ignore_units = array(
              "http://www.wikidata.org/entity/Q11573",
             "http://www.wikidata.org/entity/Q25343",
             "http://www.wikidata.org/entity/Q712226"
         );
         if(in_array($item["mainsnak"]["datavalue"]["value"]["unit"], $ignore_units)){
             $is_valid = false;
         }
     } catch (Exception $ex) {}

     if($is_valid){
         $items[] = $item;
     } else {
         $this->logger->info("Ignore value");
     }
}

@rain1024 I think the conceptBaseUri needs to be set to point to Wikidata:

$wgWBRepoSettings['conceptBaseUri'] = 'http://www.wikidata.org/entity/';

Then the units will be valid when importing. I think the default in Wikibase is to assume units are items on your local Wikibase instance and not on Wikidata.

@filbertkm I added these line to LocalSettings.php file

# WikibaseImport
$wgWBRepoSettings['conceptBaseUri'] = 'http://www.wikidata.org/entity/';
require_once "$IP/extensions/WikibaseImport/WikibaseImport.php";

But it's not working.

When I tried to import New York entity (Q1384), the console show error "Data value corrupt:upperbound filed required"

$ php maintenance/importEntities.php --entity Q1384 
...
[2017-02-03 09:47:27]: Adding statements: Q1384
[2017-02-03 09:47:31]: Data value corrupt: upperBound field required
[2017-02-03 09:47:31]: Done

Please tell me where to put these line, in LocalSettings.php file or in WikibaseImport/maintaince/importEntities.php file?

$wgWBRepoSettings['conceptBaseUri'] = 'http://www.wikidata.org/entity/';

@filbertkm Update

I add these setting in extensions/Wikibase/repo/config/Wikibase.default.php

//	'conceptBaseUri' => function() {
//		$uri = preg_replace( '!^//!', 'http://', $GLOBALS['wgServer'] );
//		return $uri . '/entity/';
//	},
    'conceptBaseUri' => 'http://www.wikidata.org/entity/',

but still have no luck

$ php maintenance/importEntities.php --entity Q1384 
...
[2017-02-03 09:47:27]: Adding statements: Q1384
[2017-02-03 09:47:31]: Data value corrupt: upperBound field required
[2017-02-03 09:47:31]: Done

I agree with the previews posts that the problem is that when there is a unit the current version is uploading it as a wikidata entry.

The code is basically doing the following. For importing the entity Q1

1. first Wikidata is checked and all objects of the statements associated to Q1 are retrieved
2. all objects are then inserted in the local wikibase
3. the statements are added to the local wikibase

In step 2 only the objects are considered. So the units are ignored and not imported. This means that for example the unit meter will not be imported in the wikibase. To avoid this I fixed the code like follows. I touched the function getReferencedEntities in src/EntityImporter.php like this:

private function getReferencedEntities( EntityDocument $entity ) {
	$statements = $entity->getStatements();
	$snaks = $statements->getAllSnaks();
	$entities = array();

	foreach( $snaks as $key => $snak ) {
		$entities[] = $snak->getPropertyId()->getSerialization();

		if ( $snak instanceof PropertyValueSnak ) {
			$value = $snak->getDataValue();
			//print_r($value);
			if ( $value instanceof EntityIdValue ) {
				$entities[] = $value->getEntityId()->getSerialization();
			}
			**if ($value instanceof UnboundedQuantityValue){
				$unit = $value->getUnit();
				if (strpos($unit, 'http://www.wikidata.org/entity/') !== false){
					$value2 = array_pop(array_reverse($this->apiEntityLookup->getEntities([str_replace("http://www.wikidata.org/entity/","",$value->getUnit())])));
					$number = $value2->getId()->getSerialization();
					$unit = $value->getUnit();
					$unit = $number;
					$entities[] = $number;
				}
			}**
		}
	}
	return array_unique( $entities );
}

Especially now the unit is extracted also from the object entities and added to the local wikibase. In step 3 the tool continues to import the entity with the old units. To avoid this I modified the function importEntities in src/EntityImporter.php and added:

if ( $importStatements === true ) {
		foreach( $stashedEntities as $entity ) {
			$referencedEntities = $this->getReferencedEntities( $entity );
			$this->importEntities( $referencedEntities, false );

			$entity_new = $entity;
			$statements_new = $entity->getStatements();
			foreach($statements_new as $key1 => $statement_new){
				$snak_new = $statement_new->getMainSnak();
				if ($snak_new instanceof PropertyValueSnak){
					print("Eneter here --------------\n");
					$data_value_new = $snak_new->getDataValue();
					print_r($qualifier_new);
					if ($data_value_new instanceof EntityIdValue){
						print("Found entity ID value \n");
					}
					if ( $data_value_new instanceof UnboundedQuantityValue ) {
						$unit = $data_value_new->getUnit();
						if (strpos($unit, 'http://www.wikidata.org/entity/') !== false){
						$id = str_replace("http://www.wikidata.org/entity/","",$unit);
						$newid = $this -> entityMappingStore->getLocalId(new ItemId($id));
						$data_value_new = new UnboundedQuantityValue($data_value_new->getAmount(),'http://YOUR_HOST/entity/'.$newid);
						$snak_new = new PropertyValueSnak($snak_new->getPropertyId(),$data_value_new);
						$statement_new -> setMainSnak($snak_new);
						$statements_new->addStatement( $statement_new,$key1);
						
						}
					}
				}
                                    $qualifiers = $statement_new->getQualifiers();
				$qualifiers_new = $statement_new->getQualifiers();
				foreach ($qualifiers as $qualifier){
					if ($qualifier instanceof PropertyValueSnak){
						$data_value = $qualifier->getDataValue();
						if ( $data_value instanceof UnboundedQuantityValue ) {
							$unit = $data_value->getUnit();
							if (strpos($unit, 'http://www.wikidata.org/entity/') !== false){
								$id = str_replace("http://www.wikidata.org/entity/","",$unit);
								$newid = $this -> entityMappingStore->getLocalId(new ItemId($id));
								$data_value_new = new UnboundedQuantityValue($data_value->getAmount(),'http://qanswer-eu.univ-st-etienne.fr/entity/'.$newid);
								$qualifier_new = new PropertyValueSnak($qualifier->getPropertyId(),$data_value_new);
								$qualifiers_new->removeSnak($qualifier);
								$qualifiers_new->addSnak($qualifier_new);
								$statement_new -> setQualifiers($qualifiers_new);
								$statements_new->addStatement( $statement_new,$key1);
							}
						}
					}	

				}
			}

			$localId = $this->entityMappingStore->getLocalId( $entity_new->getId() );

This way the old wikidata units are replaced with the new ones you created in Step 2. That's it, now you can import also entities that contain statements with units.

This should be fixed by the following pull request: https://github.com/filbertkm/WikibaseImport/pull/41