ocean icon indicating copy to clipboard operation
ocean copied to clipboard

[Integration][Github] Github-Ocean Integration

Open rubbieKelvin opened this issue 6 months ago • 3 comments

User description

Description

included repo from https://github.com/rubbieKelvin/port.io-asm

What – Added an integration for GitHub that pulls in key elements like Issues, Pull Requests, Workflows, Repositories, and Teams

Why – To give users a quick, centralized view of how their organization is using GitHub what’s being worked on, what’s in review, which workflows are running, and who’s involved.

How – Set this up by following Port’s documentation, making sure all the relevant resources were connected and synced properly.

Type of change

Please leave one option from the following and delete the rest:

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [x] New Integration (non-breaking change which adds a new integration)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Non-breaking change (fix of existing functionality that will not change current behavior)
  • [ ] Documentation (added/updated documentation)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

  • [x] Integration able to create all default resources from scratch
  • [x] Resync finishes successfully
  • [x] Resync able to create entities
  • [x] Resync able to update entities
  • [x] Resync able to detect and delete entities
  • [x] Scheduled resync able to abort existing resync and start a new one
  • [ ] Tested with at least 2 integrations from scratch
  • [ ] Tested with Kafka and Polling event listeners
  • [ ] Tested deletion of entities that don't pass the selector

Integration testing checklist

  • [x] Integration able to create all default resources from scratch
  • [x] Resync able to create entities
  • [x] Resync able to update entities
  • [x] Resync able to detect and delete entities
  • [x] Resync finishes successfully
  • [ ] If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the examples folder in the integration directory.
  • [ ] If resource kind is updated, run the integration with the example data and check if the expected result is achieved
  • [ ] If new resource kind is added or updated, validate that live-events for that resource are working as expected
  • [ ] Docs PR link here

Preflight checklist

  • [x] Handled rate limiting
  • [x] Handled pagination
  • [x] Implemented the code in async
  • [ ] Support Multi account

Screenshots

image image image image image image

API Documentation

Provide links to the API documentation used for this integration.


PR Type

Enhancement, Tests, Documentation


Description

  • Introduced new GitHub integration for Port Ocean.

    • Syncs repositories, pull requests, issues, teams, and workflows.
    • Handles GitHub API authentication, pagination, and error handling.
    • Provides webhook endpoint for GitHub events.
  • Added comprehensive type definitions and error handling for GitHub data.

  • Implemented extensive test suite for GitHub client logic.

  • Included configuration, blueprints, and documentation files.


Changes walkthrough 📝

Relevant files
Enhancement
4 files
main.py
Main integration logic for GitHub sync and webhook             
+278/-0 
client.py
GitHub API client with error handling and data fetching   
+395/-0 
types.py
Type definitions for GitHub entities                                         
+130/-0 
webhook.py
Webhook handler for GitHub event processing                           
+177/-0 
Error handling
1 files
errors.py
Custom exceptions for GitHub integration                                 
+35/-0   
Tests
1 files
test_github_client.py
Tests for GitHub client methods and error cases                   
+167/-0 
Configuration changes
6 files
blueprints.json
Blueprints for Port entities (repo, PR, issue, etc.)         
+255/-0 
port-app-config.yml
Port app resource mapping configuration                                   
+118/-0 
spec.yaml
Integration specification for Port Ocean                                 
+13/-0   
config.yaml
Integration and Port client configuration                               
+13/-0   
poetry.toml
Poetry virtualenv configuration                                                   
+3/-0     
sonar-project.properties
SonarQube project properties for code analysis                     
+2/-0     
Documentation
4 files
.env.example
Example environment variables for integration                       
+7/-0     
README.md
Basic integration readme and usage links                                 
+7/-0     
CONTRIBUTING.md
Contribution guidelines and local setup notes                       
+7/-0     
CHANGELOG.md
Changelog file for integration releases                                   
+8/-0     
Miscellaneous
2 files
Makefile
Makefile for integration build/management                               
+1/-0     
debug.py
Debug entrypoint for running the integration                         
+4/-0     
Dependencies
1 files
pyproject.toml
Poetry project configuration and dependencies                       
+110/-0 
Additional files
2 files
__init__.py [link]   
__init__.py [link]   

Need help?
  • Type /help how to ... in the comments thread for any questions about Qodo Merge usage.
  • Check out the documentation for more information.
  • rubbieKelvin avatar May 22 '25 09:05 rubbieKelvin

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
    🧪 PR contains tests
    🔒 Security concerns

    Sensitive information exposure:
    The GitHub token is stored in environment variables and used directly in the code. While this is a common practice, the token has broad access to GitHub resources. The code should implement proper scope validation to ensure the token has only the necessary permissions. Additionally, the commented-out webhook secret validation (line 27 in main.py) could allow unauthorized webhook calls if webhooks are enabled without proper secret validation.

    ⚡ Recommended focus areas for review

    Error Handling

    The webhook secret validation is commented out with a warning instead of raising an error. This could lead to security issues if webhooks are used without proper validation.

    if not os.getenv("GITHUB_WEBHOOK_SECRET"):
        # raise ValueError("GITHUB_WEBHOOK_SECRET is not set.")
        logger.warning("GITHUB_WEBHOOK_SECRET is not set. Webhook will not be enabled/usable")
    
    Pagination Handling

    Multiple comments indicate pagination is not handled, but the code fetches only the first page (100 items) of repositories, PRs, issues, etc. This could lead to incomplete data synchronization for larger GitHub organizations.

    async def get_repositories(self) -> list[GithubRepo]:
        """
        Get all repositories for the authenticated user.
        Note: This fetches the first page of repositories. For all repositories, pagination is needed.
        """
        logger.info("Fetching repositories for the authenticated user")
    
    Missing Error Handling

    The resync functions continue silently after catching exceptions for individual repositories, which could hide important errors and lead to incomplete data synchronization.

    except Exception as e:
        logger.warning(
            f"Failed to process repository {repo.get('full_name', 'unknown')}: {str(e)}"
        )
        continue
    

    qodo-merge-pro[bot] avatar May 22 '25 09:05 qodo-merge-pro[bot]

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Impact
    General
    Fix typo in comment

    Fix the typo in the comment "cehck" to "check" for better code readability and
    documentation. This helps maintain consistent and professional code quality.

    integrations/github/github/client.py [138-139]

    -# cehck for rate limit
    +# check for rate limit
     await self._handle_rate_limit_sleep(response.headers)
    
    • [ ] Apply / Chat
    Suggestion importance[1-10]: 4

    __

    Why: This suggestion corrects a minor typo in a comment, improving code readability and professionalism. While it does not affect functionality, it contributes to code quality.

    Low
    Improve comment clarity

    Replace the vague comment "err" with a more descriptive comment like "raise
    exception for HTTP errors" to improve code clarity and maintainability.

    integrations/github/github/client.py [141-142]

    -# err
    +# raise exception for HTTP errors
     response.raise_for_status()
    
    • [ ] Apply / Chat
    Suggestion importance[1-10]: 4

    __

    Why: The suggestion makes the comment more descriptive, enhancing code clarity and maintainability. This is a minor improvement that does not impact the code's behavior.

    Low
    • [ ] More

    qodo-merge-pro[bot] avatar May 22 '25 09:05 qodo-merge-pro[bot]

    This pull request is automatically being deployed by Amplify Hosting (learn more).

    Access this pull request here: https://pr-1678.d1ftd8v2gowp8w.amplifyapp.com