docs icon indicating copy to clipboard operation
docs copied to clipboard

Update the Spark/Hive integration page

Open feorlen opened this issue 1 year ago • 1 comments

This page:

https://min.io/docs/minio/linux/integrations/disaggregated-spark-and-hadoop-hive-with-minio.html

hasn't been updated in years and the actual content apparently predates its existence at this location in the docs. The information is very inaccurate:

those are for HDP 3.1.x which predates the entire CDP releases.
HDP 3.1 and CDH 6.x are equivalent to Apache Hadoop 3.1
CDP 7.1.2+ is (mostly) equivalent to Apache Hadoop 3.2
The recommendations for the Staging committer are still valid (but tuning for a workload is likely required)
CLDR does not provide any guidance for using the Magic committer and recommends staying away from it.

The Docs team doesn't currently have the needed context, so we will need to work closely with eng/customer eng for this.

feorlen avatar Jul 23 '24 14:07 feorlen

Maybe also coordinate with updates to these blog posts:

  • https://blog.min.io/migrating-hdfs-to-object-storage/
  • https://blog.min.io/migrating-from-hadoop-without-rip-and-replace/

feorlen avatar Jul 23 '24 16:07 feorlen