Filtered values not appearing/not working after selected from dropdown Post Migration to Latest SS Version

[nicmrayce]

Bug description

I'm having an issue after migrating our older Superset to the latest version 4.0.2.

The issue is that all dashboards that we've migrated from older Superset have filters non-functional. Not only the values appear null after selected (the values however, appear in dropdown), the filter no longer works too.

I even attempted to recreate the filters from scratch again but to no avail. Requiring some assistance/hints to solve the following matter. Your response is much appreciated.

How to reproduce the bug

1. Start performing migration by exporting a dashboard from any older version of Superset.
2. Then import the dashboard again in the new Superset.
3. Go to the imported dashboard.
4. Start picking any filter values from the dropdown.
5. The error will occur right after you select any particular value from the dropdown.

Screenshots/recordings

Bug #1: Bug #2:

Previous Superset version

1.5.2

Current Superset version

4.0.2

master / latest-dev

Browser

Chrome

Additional context

No response

Stack Trace

Not applicable since we're using Helm Chart Package Manager to install Superset

Checklist

• [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
• [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
• [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot[bot]]

@nicmrayce Hello there! I'm here to help you with any bugs, questions, or contributions you have for the repository. Let's work together to solve this issue.

To resolve the issue where filters on all dashboards are non-functional after migrating to the latest version of Superset, with selected values appearing as null, you need to ensure that the filter sets are correctly migrated to the new format. The provided migration script migrate_native_filters_to_new_schema.py handles this process.

Here is the relevant part of the migration script that upgrades the filter sets:

def upgrade():
    bind = op.get_bind()
    session = db.Session(bind=bind)

    dashboards = (
        session.query(Dashboard)
        .filter(Dashboard.json_metadata.like('%"native_filter_configuration"%'))
        .all()
    )
    changed_filters, changed_filter_sets = 0, 0
    for dashboard in dashboards:
        try:
            json_metadata = json.loads(dashboard.json_metadata)
            dashboard.json_metadata = json.dumps(json_metadata, sort_keys=True)

            upgrades = upgrade_dashboard(json_metadata)
            changed_filters += upgrades[0]
            changed_filter_sets += upgrades[1]
            dashboard.json_metadata = json.dumps(json_metadata, sort_keys=True)
        except Exception:
            print(f"Parsing json_metadata for dashboard {dashboard.id} failed.")
            raise

    session.commit()
    session.close()
    print(f"Upgraded {changed_filters} filters and {changed_filter_sets} filter sets.")

Additionally, you should also run the migration script cleanup erroneous parent filter IDs to ensure that any erroneous parent filter IDs are cleaned up. This script is crucial for maintaining the integrity of the filter configurations.

Here is the relevant part of the script:

def upgrade():
    bind = op.get_bind()
    session = db.Session(bind=bind)

    for dashboard in session.query(Dashboard).all():
        if dashboard.json_metadata:
            updated = False

            try:
                json_metadata = json.loads(dashboard.json_metadata)

                if filters := json_metadata.get("native_filter_configuration"):
                    filter_ids = {fltr["id"] for fltr in filters}

                    for fltr in filters:
                        for parent_id in fltr.get("cascadeParentIds", [])[:]:
                            if parent_id not in filter_ids:
                                fltr["cascadeParentIds"].remove(parent_id)
                                updated = True

                if updated:
                    dashboard.json_metadata = json.dumps(json_metadata)
            except Exception:
                logging.exception(
                    f"Unable to parse JSON metadata for dashboard {dashboard.id}"
                )

    session.commit()
    session.close()

Ensure that both migration scripts have been executed successfully. If the problem persists, you may need to manually verify that the json_metadata of your dashboards has been correctly updated according to the new format specified in the scripts ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[rusackas]

Any errors in the browser console?

[nicmrayce]

Here is the console reply. Mostly warnings.

[nicmrayce]

What do you think, @rusackas ? Do you think our team should implement the suggestions by dosubot or do you think there are other factors causing this?

[rusackas]

I honestly haven't looked into the details of it, but if Dosu's advice is correct and you're willing to open a PR, that sounds great! If there's any opportunity to add tests or simplify code along the way, that's always appreciated :D

[nicmrayce]

Okay, @rusackas , we have now received a detailed error log as well appearing on our live dashboards:

From the error, what does it deduce and could you point us to the right person in Superset that may be able to assist us more based on the error report above?

[nicmrayce]

@dosu we're using helm chart in Ruby on Rails to install this superset. Does your fix that you propose above apply in this situation?

[dosubot[bot]]

Yes, the fix I proposed applies to your situation, even when using Helm Chart in a Ruby on Rails environment. The migration scripts migrate_native_filters_to_new_schema.py and cleanup_erroneous_parent_filter_ids.py are designed to update the filter configurations in the Superset database, which is independent of the deployment method.

To address the specific error "Cannot load filter. Error: canceling statement due to conflict with recovery DETAIL: User query might have needed to see row versions that must be removed," it seems to be related to PostgreSQL's recovery process. This error typically occurs when a query conflicts with the database's recovery process, such as during a backup or replication.

Here are the steps you should follow:

1. Run the Migration Scripts: Ensure that the migration scripts migrate_native_filters_to_new_schema.py and cleanup_erroneous_parent_filter_ids.py have been executed successfully. These scripts will update the filter configurations to the new schema and clean up any erroneous parent filter IDs.

2. Check PostgreSQL Configuration: Verify that your PostgreSQL database is not in recovery mode or undergoing any maintenance that could cause conflicts with user queries. You may need to consult your database administrator for this.

3. Review Helm Chart Configuration: Ensure that your Helm Chart configuration for Superset is correctly set up, especially the database connections. Here is a snippet from the values.yaml file for reference:

   connections:
     redis_host: '{{ .Release.Name }}-redis-headless'
     redis_port: "6379"
     db_host: '{{ .Release.Name }}-postgresql'
     db_port: "5432"
     db_user: superset
     db_pass: superset
     db_name: superset
   

4. Update Superset Configuration: Make sure that the Superset configuration is correctly set up to use the new filter schema. This includes updating the json_metadata of your dashboards as per the new schema.

5. Monitor Logs: Keep an eye on the Superset logs and PostgreSQL logs for any additional errors or warnings that could provide more insight into the issue.

By following these steps, you should be able to resolve the filter functionality issues and address the PostgreSQL conflict error. If the problem persists, you may need to further investigate the specific queries causing the conflict and optimize them accordingly.

To continue talking to Dosu, mention @dosu.

[rusackas]

Sorry this isn't helpful, but changing the title a little bit from "SS" since (a) it's Superset rather than Super Set, and (b) SS has negative historical connotations (I do not mean Chevy SuperSport models)

[rusackas]

This has been silent for a loooong time, so I'll optimistically close it as stale, but let me know if this needs a revisit/reopen.