ocean icon indicating copy to clipboard operation
ocean copied to clipboard

Published 20 hours ago • verified publisher icon port-labs

Test create cloud GitHub integration implementation

Open vheckthor opened this issue 5 months ago • 3 comments

User description

Description

What - An integration to sync GitHub Cloud resources to the Port Platform, including repositories, pull requests, teams, members, and workflow runs.

Why - To allow users to import and track their GitHub Cloud resources on the Port Platform, enabling better visibility and management of their GitHub ecosystem.

How - By implementing a robust integration that:

  • Maps GitHub Cloud resources to Port concepts

  • Handles webhook events for real-time updates

  • Provides resync capabilities for all resource types

  • Supports both organization and repository-level webhooks

  • Implements proper error handling and rate limiting

  • Uses async operations for better performance

  • Includes comprehensive logging and monitoring

Type of change

  • [x] New Integration (non-breaking change which adds a new integration)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

  • [x] Integration able to create all default resources from scratch
  • [x] Resync finishes successfully
  • [x] Resync able to create entities
  • [x] Resync able to update entities
  • [x] Resync able to detect and delete entities
  • [ ] Scheduled resync able to abort existing resync and start a new one
  • [x] Tested with at least 2 integrations from scratch
  • [ ] Tested with Kafka and Polling event listeners
  • [ ] Tested deletion of entities that don't pass the selector

Integration testing checklist

  • [x] Integration able to create all default resources from scratch
  • [x] Resync able to create entities
  • [x] Resync able to update entities
  • [x] Resync able to detect and delete entities
  • [x] Resync finishes successfully
  • [x] If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the examples folder in the integration directory.
  • [x] If resource kind is updated, run the integration with the example data and check if the expected result is achieved
  • [x] If new resource kind is added or updated, validate that live-events for that resource are working as expected
  • [ ] Docs PR link here

Preflight checklist

  • [x] Handled rate limiting
  • [x] Handled pagination
  • [x] Implemented the code in async
  • [ ] Support Multi account

Screenshots

Screenshot 2025-05-23 at 2 51 31 AM (2) Screenshot 2025-05-23 at 2 51 37 AM (2) Screenshot 2025-05-23 at 2 51 43 AM (2) Screenshot 2025-05-23 at 2 51 48 AM (2)

API Documentation

  • GitHub REST API v3: https://docs.github.com/en/rest
  • GitHub Webhooks: https://docs.github.com/en/developers/webhooks-and-events/webhooks
  • GitHub GraphQL API: https://docs.github.com/en/graphql The integration supports the following resource types: Repositories Pull Requests Teams with Members Members Workflow Runs Workflow Jobs

PR Type

Enhancement, Tests, Documentation, Other, Configuration changes

Description

  • Implements a comprehensive GitHub Cloud integration for the Port Platform, enabling synchronization and management of repositories, pull requests, teams, members, and workflow runs.

  • Adds robust async clients for interacting with the GitHub Cloud REST API, including authentication, rate limiting, pagination, and resource enrichment.

  • Introduces entity processors for handling file and search references, as well as utilities for file content parsing (JSON, YAML).

  • Implements full resync logic with batch processing, enrichment options, and error handling.

  • Provides webhook event configuration, factories, and processors for real-time updates on repositories and pull requests.

  • Registers integration handlers and startup logic in the main entrypoint.

  • Adds extensive test coverage for clients, entity processors, webhook logic, and utilities using pytest fixtures and isolated mocks.

  • Includes detailed documentation: README, contributing guidelines, and changelog.

  • Adds configuration files for environment variables, Docker, Poetry, Port integration specs, blueprints, and resource mapping.

  • Provides a Makefile for development, CI workflows, and project automation.

Changes walkthrough 📝

Relevant files
Enhancement
20 files
github_client.py
New async GitHub Cloud client with high-level resource methods

integrations/github-cloud/github_cloud/clients/github_client.py

  • Implements the main async client for interacting with the GitHub Cloud
    REST API.
  • Provides high-level methods for fetching repositories, organizations,
    pull requests, issues, workflow runs/jobs, teams, and file content.
  • Supports enrichment of repository data (e.g., with languages), batch
    processing, and file content parsing.
  • Handles concurrency, error logging, and file reference resolution.
    		• +574/-0 
    rest_client.py
    REST client for paginated GitHub Cloud API resources         

    integrations/github-cloud/github_cloud/clients/rest_client.py

  • Implements a REST client for paginated GitHub Cloud API resources.
  • Provides methods for fetching organization/repository resources, file
    content, and repo languages.
  • Handles pagination, response parsing, and content decoding (including
    base64).
  • Includes utility for parsing pagination links from response headers.
    		• +244/-0 
    base_client.py
    Base HTTP client for GitHub Cloud API with rate limiting 

    integrations/github-cloud/github_cloud/clients/base_client.py

  • Introduces a base HTTP client for GitHub Cloud API with async request
    handling.
  • Handles authentication, rate limiting, and error management.
  • Provides a method for parsing pagination links from response headers.
    		• +112/-0 
    client_factory.py
    Singleton factory for GitHubCloudClient creation                 

    integrations/github-cloud/github_cloud/clients/client_factory.py

  • Implements a singleton factory for creating and caching a
    GitHubCloudClient instance.
  • Retrieves configuration from the Port Ocean context.
    		• +24/-0   
    utils.py
    Utilities for object kinds and file content parsing           

    integrations/github-cloud/github_cloud/helpers/utils.py

  • Defines the ObjectKind enum for supported GitHub resource types.
  • Adds robust file content parsing utility for JSON and YAML, with
    logging and fallback.
  • Provides helper functions for parsing file content in various formats.
    		• +119/-0 
    entity_processor.py
    Entity processors for file and search references in entities

    integrations/github-cloud/github_cloud/entity_processors/entity_processor.py

  • Implements entity processors for handling file:// and search://
    references in entities.
  • Supports fetching file content and checking file existence in
    repositories.
  • Includes error handling and repository info extraction logic.
    		• +165/-0 
    resync_data.py
    Async resync logic for GitHub resources with enrichment   

    integrations/github-cloud/github_cloud/resync_data.py

  • Implements async resync logic for repositories, pull requests, teams
    with members, and members.
  • Supports batch processing, enrichment, and configuration-driven
    options (e.g., include_languages, include_bot_members).
  • Handles error logging and selector configuration extraction.
    		• +215/-0 
    events.py
    Event configuration classes for webhook event selection   

    integrations/github-cloud/github_cloud/webhook/events.py

  • Defines abstract and concrete event configuration classes for
    repository and organization webhooks.
  • Uses dataclasses for event configuration and provides dictionary
    conversion for API payloads.
    		• +126/-0 
    _base_webhook_factory.py
    Abstract base class for GitHub webhook factories                 

    integrations/github-cloud/github_cloud/webhook/webhook_factory/_base_webhook_factory.py

  • Implements an abstract base class for webhook factories.
  • Provides logic for webhook creation, existence checks, payload
    building, and response validation.
  • Supports generic event configuration types.
    		• +145/-0 
    repository_webhook_factory.py
    Repository webhook factory for GitHub Cloud integration   

    integrations/github-cloud/github_cloud/webhook/webhook_factory/repository_webhook_factory.py

  • Implements a factory for creating repository webhooks.
  • Handles webhook creation for multiple repositories and event
    configuration.
  • Includes error handling and logging for batch operations.
    		• +123/-0 
    organization_webhook_factory.py
    Organization webhook factory for GitHub Cloud integration

    integrations/github-cloud/github_cloud/webhook/webhook_factory/organization_webhook_factory.py

  • Implements a factory for creating organization webhooks.
  • Handles batch creation for all accessible organizations with error
    handling.
  • Supports event configuration and logging.
    		• +122/-0 
    _github_abstract_webhook_processor.py
    Abstract webhook processor for GitHub Cloud events             

    integrations/github-cloud/github_cloud/webhook/webhook_processors/_github_abstract_webhook_processor.py

  • Implements an abstract webhook processor for GitHub Cloud events.
  • Provides common logic for event type extraction, authentication, and
    payload validation.
  • Supports event filtering and logging.
    		• +115/-0 
    repository_webhook_processor.py
    Repository webhook processor for GitHub Cloud events         

    integrations/github-cloud/github_cloud/webhook/webhook_processors/repository_webhook_processor.py

  • Implements a webhook processor for repository events.
  • Handles creation, update, and deletion events for repositories.
  • Extracts repository data and manages result formatting.
    		• +101/-0 
    pull_request_webhook_processor.py
    Pull request webhook processor for GitHub Cloud events     

    integrations/github-cloud/github_cloud/webhook/webhook_processors/pull_request_webhook_processor.py

  • Implements a webhook processor for pull request events.
  • Handles creation, update, and closure events for pull requests.
  • Extracts pull request data and manages result formatting.
    		• +136/-0 
    main.py
    Main integration entrypoint and handler registration         

    integrations/github-cloud/main.py

  • Implements the main integration entrypoint for Port Ocean.
  • Registers resync handlers for repositories, pull requests, teams, and
    members.
  • Handles webhook processor registration and startup logic for webhook
    creation.
  • Includes error handling and logging throughout.
    		• +191/-0 
    Makefile
    Add Makefile for development and CI workflows                       

    integrations/github-cloud/Makefile

  • Added a Makefile with commands for install, test, lint, format, clean,
    Docker, and more.
  • Included targets for development, release, documentation, and security
    checks.
  • Provided a help target listing available commands.
    		• +101/-0 
    auth_client.py
    Add AuthClient for GitHub API authentication                         

    integrations/github-cloud/github_cloud/clients/auth_client.py

  • Implemented AuthClient class for GitHub Cloud authentication.
  • Provided method to generate authentication headers for API requests.
  • Included docstrings for clarity.
    		• +23/-0   
    __init__.py
    Add clients package initialization and exports                     

    integrations/github-cloud/github_cloud/clients/init.py

  • Added __init__.py to expose AuthClient, HTTPBaseClient,
    GitHubCloudClient, and RestClient.
  • Defined __all__ for explicit module exports.
    		• +6/-0     
    __init__.py
    Add entity_processors package initialization and exports 

    integrations/github-cloud/github_cloud/entity_processors/init.py

  • Added __init__.py to expose FileEntityProcessor and
    SearchEntityProcessor.
  • Defined __all__ for explicit module exports.
    		• +6/-0     
    __init__.py
    Add webhook_factory package initialization and exports     

    integrations/github-cloud/github_cloud/webhook/webhook_factory/init.py

  • Added __init__.py to expose BaseWebhookFactory,
    RepositoryWebhookFactory, and OrganizationWebhookFactory.
  • Defined __all__ for explicit module exports.
    		• +9/-0     
    Miscellaneous
    1 files
    __init__.py
    Init file for webhook processor module exports                     

    integrations/github-cloud/github_cloud/webhook/webhook_processors/init.py

  • Adds an __init__ file to expose webhook processor classes.
  • Imports and lists abstract, repository, and pull request processors.
    		• +9/-0     
    Tests
    9 files
    conftest.py
    Pytest fixtures for Port Ocean and HTTP client mocking     

    integrations/github-cloud/tests/conftest.py

  • Provides pytest fixtures for mocking Port Ocean context and HTTP
    client.
  • Ensures isolation and mocking for all test modules.
    		• +24/-0   
    test_clients.py
    Tests for AuthClient, HTTPBaseClient, and client factory 

    integrations/github-cloud/tests/test_clients.py

  • Adds tests for AuthClient, HTTPBaseClient, and client factory logic.
  • Covers authentication headers, API request handling, rate limiting,
    and singleton behavior.
    		• +109/-0 
    test_entity_processors.py
    Tests for file and search entity processors                           

    integrations/github-cloud/tests/test_entity_processors.py

  • Adds tests for FileEntityProcessor and SearchEntityProcessor.
  • Covers file content fetching, search pattern parsing, and error
    handling.
    		• +127/-0 
    test_github_client.py
    Tests for GitHubCloudClient high-level methods                     

    integrations/github-cloud/tests/test_github_client.py

  • Adds tests for the main GitHubCloudClient methods.
  • Covers repository, organization, pull request, issue, workflow, and
    team member fetching.
  • Tests enrichment, file content, and search logic.
    		• +157/-0 
    test_rest_client.py
    Tests for RestClient pagination and file/language methods

    integrations/github-cloud/tests/test_rest_client.py

  • Adds tests for the RestClient's paginated resource fetching and
    file/language methods.
  • Covers pagination, file content decoding, and search API handling.
    		• +175/-0 
    test_utils.py
    Tests for ObjectKind enum and file content parsing             

    integrations/github-cloud/tests/test_utils.py

  • Adds tests for ObjectKind enum and file content parsing utility.
  • Covers JSON, YAML, multi-document YAML, and fallback scenarios.
    		• +105/-0 
    test_webhook_factory.py
    Tests for repository and organization webhook factories   

    integrations/github-cloud/tests/test_webhook_factory.py

  • Adds tests for repository and organization webhook factories.
  • Covers webhook creation, existence checks, event configuration, and
    error handling.
    		• +169/-0 
    test_webhook_processors.py
    Tests for repository and pull request webhook processors 

    integrations/github-cloud/tests/test_webhook_processors.py

  • Adds tests for repository and pull request webhook processors.
  • Covers authentication, event filtering, payload validation, and event
    handling logic.
  • Tests edge cases and error handling for webhook event processing.
    		• +263/-0 
    test_webhook_events.py
    Add tests for webhook event configuration classes               

    integrations/github-cloud/tests/test_webhook_events.py

  • Added tests for RepositoryEvents and OrganizationEvents classes.
  • Tested default and custom values, to_dict methods, and immutability.
  • Used pytest and dataclasses for assertions and error checks.
    		• +91/-0   
    Documentation
    3 files
    README.md
    Initial project README with installation and usage instructions

    integrations/github-cloud/README.md

  • Added a comprehensive README file describing the GitHub Cloud
    integration.
  • Included installation, development, and contribution instructions.
  • Provided an overview of features and project structure.
  • Linked to documentation and license information.
    		• +89/-0   
    CONTRIBUTING.md
    Add contributing guidelines reference file                             

    integrations/github-cloud/CONTRIBUTING.md

  • Added a CONTRIBUTING.md file with a link to the integration
    contributing guide.
    		• +3/-0     
    CHANGELOG.md
    Add changelog file for project version tracking                   

    integrations/github-cloud/CHANGELOG.md

  • Added a CHANGELOG.md file to document notable project changes.
  • Adopted Keep a Changelog and Semantic Versioning standards.
    		• +6/-0     
    Configuration changes
    8 files
    .env.example
    Provide example environment configuration file                     

    integrations/github-cloud/.env.example

  • Added an example environment file with required environment variables
    for configuration.
  • Included placeholders for GitHub token and Port integration settings.
    		• +7/-0     
    poetry.toml
    Add Poetry configuration for in-project virtualenvs           

    integrations/github-cloud/poetry.toml

  • Added poetry.toml to configure Poetry to use in-project virtual
    environments.
    		• +2/-0     
    pyproject.toml
    Add project and dependency configuration with pyproject.toml

    integrations/github-cloud/pyproject.toml

  • Added pyproject.toml for project metadata and dependency management.
  • Defined dependencies, development tools, and configuration for mypy,
    ruff, black, and pytest.
  • Configured towncrier for changelog management.
    		• +112/-0 
    Dockerfile
    Add Dockerfile for containerized deployment                           

    integrations/github-cloud/Dockerfile

  • Added a Dockerfile for building and running the integration in a
    container.
  • Configured Poetry installation and dependency management.
  • Set up the entrypoint to run the application.
    		• +32/-0   
    .dockerignore
    Add .dockerignore to optimize Docker builds                           

    integrations/github-cloud/.dockerignore

  • Added a .dockerignore file to exclude unnecessary files and
    directories from Docker build context.
  • Excluded VCS, virtual environments, test, documentation, and cache
    files.
    		• +78/-0   
    spec.yaml
    Add Port integration specification file                                   

    integrations/github-cloud/.port/spec.yaml

  • Added Port integration specification YAML describing supported
    features and configuration options.
  • Defined exporter resources and required configuration fields.
    		• +22/-0   
    blueprints.json
    Add Port blueprints for GitHub resource mapping                   

    integrations/github-cloud/.port/resources/blueprints.json

  • Added blueprints.json defining Port blueprints for GitHub resources
    (Repository, Member, Team, Pull Request).
  • Specified schemas, properties, and relations for each blueprint.
    		• +157/-0 
    port-app-config.yml
    Add Port app configuration for resource mapping                   

    integrations/github-cloud/.port/resources/port-app-config.yml

  • Added Port app configuration YAML mapping GitHub resources to Port
    entities.
  • Defined selectors, entity mappings, properties, and relations for
    repository, member, team, and pull request.
    		• +69/-0   
    Additional files
    3 files
    __init__.py [link]   
    __init__.py [link]   
    __init__.py [link]   
    Need help?
  • Type /help how to ... in the comments thread for any questions about Qodo Merge usage.
  • Check out the documentation for more information.

    • vheckthor avatar May 23 '25 02:05 vheckthor