Some Modifications on Distributed Rendering

[hw4270]

When I used distributed rendering, I found no difference from normal rendering. So I debugged the source code and found some problems in the official package. After I modified some of the source code, the map was displayed properly. I don't know if this is the best way to modify it, but it worked. The package I used are "geowave-hbase-0.9.8-apache.jar" and "geowave-geoserver-0.9.8-apache.jar". Here are some of my modifications.

1.The file "\extensions\adapters\vector\src\main\java\applicationContext.xml" was lost in the package "geowave-geoserver-0.9.8-apache.jar". So distributed rendering doesn't work.

<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<bean id="getMapCallback" class="mil.nga.giat.geowave.adapter.vector.render.DistributedRenderCallback">
<constructor-arg ref="wms" />
</bean>
</beans>

2.Class "mil.nga.giat.geowave.adapter.vector.render.DistributedRenderCallback", Method "beforeLayer". I think this should be "style.getTransformation() instanceof ProcessFunction ". The condition "style instanceof ProcessFunction" is always false.

// if (style instanceof ProcessFunction && style.getTransformation() != null
if (style != null && style.getTransformation() != null
	&& (((ProcessFunction) style.getTransformation()).getName() != null)
	&& ((ProcessFunction) style.getTransformation()).getName().equals(
	 DistributedRenderProcess.PROCESS_NAME))

3.Class "mil.nga.giat.geowave.adapter.vector.render.InternalDistributedRenderProcess", Rendering is repeated here.

if (features != null) {
	final SimpleFeatureIterator it = features.features();
	if (it.hasNext()) {
		final SimpleFeature resultFeature = it.next();  //features.features().next();

4.Class "mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveFeatureSource", Added code

@Override
protected boolean canRetype() {
	return true;
}

5.Class "mil.nga.giat.geowave.datastore.hbase.operations.HBaseOperations",Method "aggregateServerSide", Added a line of code to set InternalAdapterId.

if (readerParams.getAggregation().getLeft() != null) {
  if (readerParams.getAggregation().getRight() instanceof CommonIndexAggregation) {...}
  else if (readerParams.getAggregation().getLeft() instanceof InternalDataAdapterWrapper) {
      InternalDataAdapterWrapper wrapper = ... 
      requestBuilder.setAdapter(ByteString.copyFrom(URLClassloaderUtils.toBinary(wrapper.getAdapter())));           
      requestBuilder.setInternalAdapterId(ByteString.copyFrom(ByteArrayUtils.shortToByteArray(wrapper
							.getInternalAdapterId()))); // added
  }
  else {...}
}

6.Class "mil.nga.giat.geowave.datastore.hbase.coprocessors.AggregationEndpoint", Method"aggregate". All of adapters will be displayed if not set internalAdapterId.

if (request.hasAdapter()) {
	final byte[] adapterBytes = request.getAdapter().toByteArray();
	dataAdapter = (DataAdapter) URLClassloaderUtils.fromBinary(adapterBytes);
}
//else if (request.hasInternalAdapterId()) {
if (request.hasInternalAdapterId()) {
	final byte[] adapterIdBytes = request.getInternalAdapterId().toByteArray();
	internalAdapterId = ByteArrayUtils.byteArrayToShort(adapterIdBytes);
}

7.Class "mil.nga.giat.geowave.adapter.vector.render.DistributedRenderAggregation", Method "initRenderer". The time here should be milliseconds, while "options.getMaxRenderTime()" is in seconds.

//formatOptions.put("timeout", options.getMaxRenderTime());
formatOptions.put("timeout", options.getMaxRenderTime() * 1000);

I have 2.12 million polygon features, the map display takes 7 seconds. It seems that everything is all right. But when the data exceeds 10 million, the region servers dead. This is the next problem to be solved. Can someone help me？ @rfecher

[rfecher]

Thats great that you were able to work through it. I have it working in some deployments but our integration tests are checking correctness when using this SLD. The challenge is ensuring it is in fact rendering it distributed. I'll have to double check the current IT and look through your suggestions when I have a chance. Thanks for the feedback!

Re: the region servers dying on 10 million polys, I'd be happy to try to help - is there anything in the region server's logs that might be relevant?

[hw4270]

Yes, there are some warnings in the log, but I don't know what's wrong.

2019-03-11 14:30:51,504 WARN [RpcServer.default.FPBQ.Fifo.handler=193,queue=13,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"ExecService(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$CoprocessorServiceRequest)","starttimems":1552285653329,"responsesize":7327,"method":"ExecService","param":"coprocessorService=AggregationService:aggregate","processingtimems":198175,"client":"192.168.10.24:29659","queuetimems":0,"class":"HRegionServer"}

2019-03-11 14:31:05,217 WARN
[RpcServer.default.FPBQ.Fifo.handler=197,queue=17,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"ExecService(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$CoprocessorServiceRequest)","starttimems":1552285667533,"responsesize":12533,"method":"ExecService","param":"coprocessorService=AggregationService:aggregate","processingtimems":197684,"client":"192.168.10.24:29659","queuetimems":0,"class":"HRegionServer"}

2019-03-11 14:32:06,951 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=521.11 MB, freeSize=27.44 MB, max=548.55 MB, blockCount=8179, accesses=39755881, hits=22147798, hitRatio=55.71%, , cachingAccesses=20851029, cachingHits=19913807, cachingHitsRatio=95.51%, evictions=44830, evicted=906365, evictedPerRun=20.21782288645996

[rfecher]

one thing I can suggest trying is adding more splits. When you setup your geowave index you can choose numPartitions which will automatically give you more randomized splits and may allow for a single region to complete the aggregation in sufficient time (although there will be more regions so the total time will likely be similar). One other thing I had considered was to enable aggregation either within the coprocessor or within spark. Would aggregating (in this case distributed rendering) in spark be a good option in your case?

[hw4270]

Thank you for your suggestions, I'll try to use aggregation in spark.

[rfecher]

no prob - if you have a contribution you think would be generally useful (distributed rendering in spark could definitely qualify), PRs are always appreciated 😃

[hw4270]

Well, if I make any progress, I'll give you feedback.

[MuhammedKalkan]

@rfecher @hw4270 Hello, I was trying to read more about distributed rendering for days but , can not find any info anywhere except this issue keeps popping up.

Can you please give some info about it like :

1- There is an DistributedRendering.sld in examples folder, but how do we use it ? Is it even relevant ? 2- Is distributed rendering only applicable to Accumulo and HBase ? For ex Cassandra and Redis not available for distributed rendering ? 3- Do we need to install some additional jars / applications on data nodes to make distributed rendering work ? 4- How does Geoserver tells nodes to render data and how do nodes understand it ? (Any client apps on nodes ?) 5- Do we need to install any other tools / software / setup except GeoServer, GeoServer-geowave plugin and some datastore ? 6- Can you supply some example project / data for me to work on and understand it ?

Best regards and sorry for the inconvenience

[rfecher]

1. yes, that example SLD is how you'd use it through GeoServer (in particular it is enabled through the <Transformation> element defining a geowave:DistributedRender render transform)
2. yes, although it will "work" with any data store but the only time the processing is truly distributed is when you have server-side processing which is the case for HBase (through Coprocessors) and Accumulo (through Iterators) only at the moment
3. yes, both Accumulo and HBase have a concept of a distributed classpath and a particular geowave jar needs to be available (the jars are appropriately marked geowave-accumulo and geowave-hbase in the downloads section). We also have RPMs for each which automatically place the appropriate jar on the distributed file system. Theer should be detail instructions available.
4. No "special" client apps on nodes other than of course what is already in the stack for the datastore you're using (ie. HBase region server or Accumulo tablet server). GeoWave has a built-in concept of an Aggregation which takes advantage of server-side processing and can be anything from computing counts or bounding boxes for a given query to distributed rendering. The render transform above triggers geowave to use a distributed rendering aggregation.
5. No
6. The example Distributed Rendering SLD should suffice. Take the Vector Demo as an example but instead of using the SubsamplePoints SLD use the Distributed Rendering SLD.

[scially]

When I used this SLD, I found that the CPU of the RegionServer was all 0%. . .