sparkmagic displays pandas DataFrames as texts

[sergun]

Hi!

A have an issue with displaying pandas DataFrame in sparkmagic. They are always displayed as text.

Any ideas?

sparkmagic 0.12.6 livy 0.5

[bakousylla]

      Hi!

A have an issue with displaying pandas DataFrame in sparkmagic. They are always displayed as text. Any ideas? sparkmagic 0.12.6 livy 0.5

Hi! I have exactly the same issue with plotly, did you fix that or have any ideas ? sparkmagic 0.12.6

[Bootcampanalytics]

Hi, same here, and I'd love a fix!!

[itamarst]

Have you tried the latest version, 0.12.9?

[Bootcampanalytics]

Have you tried the latest version, 0.12.9?

Yes, this is the version that AWS uses today.

I posted this here: https://stackoverflow.com/questions/57794930/pandas-formatting-in-emr-notebooks-in-cluster-mode-or-transferring-to-local-with

[devender-yadav]

@analytics-bootcamp I doubt it if it's EMR. EMR notebook is using 0.12.7 as of EMR 5.26 version.

[Bootcampanalytics]

@analytics-bootcamp I doubt it if it's EMR. EMR notebook is using 0.12.7 as of EMR 5.26 version.

This is what I did:

In AWS, I start a cluster and open an EMR notebook.

import sparkmagic as sm. -> error.

next try: %local import sparkmagic as sm print (sm.__version__)

-> 0.12.9

[devender-yadav]

Looks like they updated it in last 1-2 days. I noticed progress bar in EMR notebook today.

[itamarst]

Can you give me an easy way to reproduce this? Thanks!

[jwillis0720]

@itamarst

spark_df = spark.read.parquet("s3://path_to_parquet")
pandas_df = spark_df.limit(100).to_pandas()
pandas_df
> shows text instead of html markup in emr jupyter lab or jupyter notebook