Allow multiple integration tests at the main app level

[josecelano]

I've added an integration test for global metrics:

https://github.com/torrust/torrust-tracker/issues/1407

All tests except this one are inside the packages. However, this test if for a behavior that you can only test at the main app level.

You can run multiple trackers (HTTP and UDP) on different socket addresses and the metrics in the REST API endpoint /api/v1/stats returns the global metrics. That mean counters, for example for requests, include requests to all the trackers.

We might want to write integration tests for other things as integration tests.

We also have E2E tests that use docker to run the application in a independent process. However, integration tests:

• Are faster
• Allow to change the context for the test (for example configuration)
• You can run them everywhere (like docker image build)

You can run the sample integration test with:

$ cargo test --test integration
    Finished `test` profile [optimized + debuginfo] target(s) in 0.14s
     Running tests/integration.rs (target/debug/deps/integration-3c1f7d685bf9a56c)

running 1 test
test servers::api::contract::stats::the_stats_api_endpoint_should_return_the_global_stats ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

~~Problem 1: The logging in the configuration is disabled~~

UPDATE: See the comment below.

[logging]
threshold = "off"

It can be enabled now that there is only one test. However, if you add a second test that also tries to setup the logging you would get an error because it tries to set a global subscriber twice.

Problem 2: You have to use different ports for each tests

This was also a problem we had for other tests. If you use the same configuration with pre-defined ports you are going to get error because you can't listen to an address that's already in use. The error looks like:

Could not bind tcp_listener to address.: Os { code: 98, kind: AddrInUse, message: "Address already in use" }

You can change the port but it's paint to control port in use. We solved for other tests using port 0. The system assign free random port. However, you need to capture the assigned port before running your requests in the asserts.

In the E2E tests (black box) we parse the output to get the IPs and ports.

Now, there is an alternative that might be better for integration tests. I've added the "listen url" the Registar so it's available directly via the Registar type.

Problem 3: You can't inject configuration via env var

This is also another problem we had when we started adding tests. If we inject the configuration with an env var you will have unexpected behavior because tests run in parallel and one test configuration can be overwritten by other tests.

We solved in other cases by injecting the configuration via a file in a temporary directory.

Conclusion

Other problem might arise when we add a second integration test. I din't fix them because:

• I'm focused right now on the stats overhaul and I only want to make sure I don't break global metrics when running multiple trackers.
• We should not add many tests at this level. Most of the tests should be in the corresponding package regardless they are unit, integration or E2E tests. At this level we should test only the main app logic: jobs, app container, bootstrapping, initialization of services, running multiple trackers, or in general logic related to having multiple services.

For now, I only wanted to document these problems to make it easier to fix them when we need to add more integration tests at the main app level.

[josecelano]

Hi @da2ce7 I can't reproduce problem 1 anymore, so I have removed it from the list for now.

I tried to reproduce it by replacing the whole integration test filetests/servers/api/contract/stats/mod.rs with:

use std::env;

use torrust_tracker_lib::app;

#[tokio::test]
async fn the_stats_api_endpoint_should_return_the_global_stats_1() {
    let config_with_two_http_trackers = r#"
        [metadata]
        app = "torrust-tracker"
        purpose = "configuration"
        schema_version = "2.0.0"

        [logging]
        threshold = "info"

        [core]
        listed = false
        private = false

        [core.database]
        driver = "sqlite3"
        path = "./integration_tests_sqlite3.db"

        [health_check_api]
        bind_address = "0.0.0.0:0"        
        "#;

    env::set_var("TORRUST_TRACKER_CONFIG_TOML", config_with_two_http_trackers);

    let (_app_container, _jobs, _registar) = app::run().await;
}

#[tokio::test]
async fn the_stats_api_endpoint_should_return_the_global_stats_2() {
    let config_with_two_http_trackers = r#"
        [metadata]
        app = "torrust-tracker"
        purpose = "configuration"
        schema_version = "2.0.0"

        [logging]
        threshold = "info"

        [core]
        listed = false
        private = false

        [core.database]
        driver = "sqlite3"
        path = "./integration_tests_sqlite3.db"

        [health_check_api]
        bind_address = "0.0.0.0:0"        
        "#;

    env::set_var("TORRUST_TRACKER_CONFIG_TOML", config_with_two_http_trackers);

    let (_app_container, _jobs, _registar) = app::run().await;
}

• By using port 0 we don't have problem 2.
• By using the same config env var value, you don't have problem 3

The output is fine:

$ cargo test -- --nocapture 
   Compiling torrust-tracker v3.0.0-develop (/home/josecelano/Documents/git/committer/me/github/torrust/torrust-tracker)
    Finished `test` profile [optimized + debuginfo] target(s) in 41.40s
     Running unittests src/lib.rs (target/debug/deps/torrust_tracker_lib-b7b2d1fc55bfe6d2)

running 8 tests
Loading extra configuration from default configuration file: `./share/default/config/tracker.development.sqlite3.toml` ...
test console::ci::e2e::logs_parser::tests::it_should_replace_wildcard_ip_with_localhost ... ok
test bootstrap::config::tests::it_should_load_with_default_config ... ok
test console::ci::e2e::logs_parser::tests::it_should_ignore_logs_with_no_matching_lines ... ok
test console::ci::e2e::logs_parser::tests::it_should_parse_from_logs_with_valid_logs ... ok
test console::ci::e2e::logs_parser::tests::it_should_parse_multiple_services ... ok
test console::ci::e2e::logs_parser::tests::it_should_support_colored_output ... ok
test bootstrap::jobs::http_tracker::tests::it_should_start_http_tracker ... ok
test bootstrap::jobs::tracker_apis::tests::it_should_start_http_tracker ... ok

test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/bin/e2e_tests_runner.rs (target/debug/deps/e2e_tests_runner-f6f80a69d5eb03c8)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/bin/http_health_check.rs (target/debug/deps/http_health_check-f1a2a0212e5183ec)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/bin/profiling.rs (target/debug/deps/profiling-8fb61b87807ca189)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/torrust_tracker-91a953124cd232d6)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/integration.rs (target/debug/deps/integration-015c9121a9272ff8)

running 2 tests
Loading extra configuration from environment variable:
 
        [metadata]
        app = "torrust-tracker"
        purpose = "configuration"
        schema_version = "2.0.0"

        [logging]
        threshold = "info"

        [core]
        listed = false
        private = false

        [core.database]
        driver = "sqlite3"
        path = "./integration_tests_sqlite3.db"

        [health_check_api]
        bind_address = "0.0.0.0:0"        
        
Loading extra configuration from default configuration file: `./share/default/config/tracker.development.sqlite3.toml` ...
Loading extra configuration from environment variable:
 
        [metadata]
        app = "torrust-tracker"
        purpose = "configuration"
        schema_version = "2.0.0"

        [logging]
        threshold = "info"

        [core]
        listed = false
        private = false

        [core.database]
        driver = "sqlite3"
        path = "./integration_tests_sqlite3.db"

        [health_check_api]
        bind_address = "0.0.0.0:0"        
        
Loading extra configuration from default configuration file: `./share/default/config/tracker.development.sqlite3.toml` ...
2025-04-04T09:56:36.115216Z  INFO torrust_tracker_configuration::logging: Logging initialized
2025-04-04T09:56:36.115232Z  INFO torrust_tracker_lib::bootstrap::app: Configuration:
{
  "metadata": {
    "app": "torrust-tracker",
    "purpose": "configuration",
    "schema_version": "2.0.0"
  },
  "logging": {
    "threshold": "info"
  },
  "core": {
    "announce_policy": {
      "interval": 120,
      "interval_min": 120
    },
    "database": {
      "driver": "sqlite3",
      "path": "./integration_tests_sqlite3.db"
    },
    "inactive_peer_cleanup_interval": 600,
    "listed": false,
    "net": {
      "external_ip": "0.0.0.0",
      "on_reverse_proxy": false
    },
    "private": false,
    "private_mode": null,
    "tracker_policy": {
      "max_peer_timeout": 900,
      "persistent_torrent_completed_stat": false,
      "remove_peerless_torrents": true
    },
    "tracker_usage_statistics": true
  },
  "udp_trackers": null,
  "http_trackers": null,
  "http_api": null,
  "health_check_api": {
    "bind_address": "0.0.0.0:0"
  }
}
2025-04-04T09:56:36.115238Z  INFO torrust_tracker_lib::bootstrap::app: Configuration:
{
  "metadata": {
    "app": "torrust-tracker",
    "purpose": "configuration",
    "schema_version": "2.0.0"
  },
  "logging": {
    "threshold": "info"
  },
  "core": {
    "announce_policy": {
      "interval": 120,
      "interval_min": 120
    },
    "database": {
      "driver": "sqlite3",
      "path": "./integration_tests_sqlite3.db"
    },
    "inactive_peer_cleanup_interval": 600,
    "listed": false,
    "net": {
      "external_ip": "0.0.0.0",
      "on_reverse_proxy": false
    },
    "private": false,
    "private_mode": null,
    "tracker_policy": {
      "max_peer_timeout": 900,
      "persistent_torrent_completed_stat": false,
      "remove_peerless_torrents": true
    },
    "tracker_usage_statistics": true
  },
  "udp_trackers": null,
  "http_trackers": null,
  "http_api": null,
  "health_check_api": {
    "bind_address": "0.0.0.0:0"
  }
}
2025-04-04T09:56:36.116667Z  WARN start: torrust_tracker_lib::app: No services enabled in configuration
2025-04-04T09:56:36.116675Z  INFO start: torrust_tracker_lib::app: No UDP blocks in configuration
2025-04-04T09:56:36.116667Z  WARN start: torrust_tracker_lib::app: No services enabled in configuration
2025-04-04T09:56:36.116685Z  INFO start: torrust_tracker_lib::app: No UDP blocks in configuration
2025-04-04T09:56:36.116679Z  INFO start: torrust_tracker_lib::app: No HTTP blocks in configuration
2025-04-04T09:56:36.116693Z  INFO start: torrust_tracker_lib::app: No API block in configuration
2025-04-04T09:56:36.116688Z  INFO start: torrust_tracker_lib::app: No HTTP blocks in configuration
2025-04-04T09:56:36.116702Z  INFO start: torrust_tracker_lib::app: No API block in configuration
2025-04-04T09:56:36.116731Z  INFO HEALTH CHECK API: Starting on: http://0.0.0.0:0
2025-04-04T09:56:36.116732Z  INFO HEALTH CHECK API: Starting on: http://0.0.0.0:0
2025-04-04T09:56:36.116879Z  INFO start:start_job: HEALTH CHECK API: Started on: http://0.0.0.0:40813
2025-04-04T09:56:36.116879Z  INFO start:start_job: HEALTH CHECK API: Started on: http://0.0.0.0:43429
test servers::api::contract::stats::the_stats_api_endpoint_should_return_the_global_stats_1 ... ok
test servers::api::contract::stats::the_stats_api_endpoint_should_return_the_global_stats_2 ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests torrust_tracker_lib

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

I guess the unique initialization of the logging is working fine in this case. I don't remember why we had a problem with this in the past in other tests or why I got the error "duplication initialization of the global subscriber" while I was working on the integration test. Anyway, for now, I have removed the problem from the list.