magma feat(agw): Write per subscriber APN specific data into subscriberDB store when MME receives Update Location Answer

Summary

This PR has the following new functions:

SQLite functions that allow adding subscribers
A function to convert ula response to subscriber data which is then written into the sqlite database

Test Plan

Unit tests for the function that converts ula to sub_data Test that an attach/detach is still successful

Additional Information

[ ] This change is backwards-breaking

Aug 01 '22 21:08 vroon2703

Thanks for opening a PR! :100:

A couple initial guidelines

All commits must be signed off. This is enforced by DCO check.
All PR titles must follow the semantic commits format. This is enforced by Semantic PR.

Howto

Reviews. The "Reviewers" listed for this PR are the Magma maintainers who will shepherd it.
Checks. All required CI checks must pass before merge.
Merge. Once approved and passing CI checks, use the ready2merge label to indicate the maintainers can merge your PR.

More info

Please take a moment to read through the Magma project's

Contributing Conventions for norms around contributed code

If this is your first Magma PR, also consider reading

Developer Onboarding for onboarding as a new Magma developer
Development Workflow for guidance on your first pull request
CI Checks for points of contact for failing or flaky CI checks
GitHub-to-Slack mappings for Magma maintainers for guidance on how to contact maintainers on Slack

Aug 01 '22 21:08 github-actions[bot]

Oops! Looks like you failed the Python Format Check.

Howto

Instructions on running the formatter and linter locally are provided in the format AGW doc
Guide to the different CI checks and resolution guidelines

Aug 01 '22 21:08 github-actions[bot]

feg-workflow

    2 files 203 suites 40s :stopwatch: 374 tests 374 :heavy_check_mark: 0 :zzz: 0 :x: 388 runs 388 :heavy_check_mark: 0 :zzz: 0 :x:

Results for commit 29341fc6.

:recycle: This comment has been updated with latest results.

Aug 01 '22 21:08 github-actions[bot]

dp-workflow

13 tests 13 :heavy_check_mark: 2m 8s :stopwatch:   1 suites   0 :zzz:   1 files   0 :x:

Results for commit 29341fc6.

:recycle: This comment has been updated with latest results.

Aug 01 '22 22:08 github-actions[bot]

agw-workflow

615 tests 611 :heavy_check_mark: 4m 56s :stopwatch:     2 suites     4 :zzz:     2 files     0 :x:

Results for commit 29341fc6.

:recycle: This comment has been updated with latest results.

Aug 01 '22 22:08 github-actions[bot]

cloud-workflow

1 011 tests 1 011 :heavy_check_mark: 2m 12s :stopwatch:   356 suites       0 :zzz:       7 files       0 :x:

Results for commit 29341fc6.

:recycle: This comment has been updated with latest results.

Aug 01 '22 22:08 github-actions[bot]

@vroon2703 , switching the default return values to "false" for GetCloudSubscriberDbEnabled should fix the cloud tests.

Aug 03 '22 05:08 ssanadhya

The Bazel build fails, because there is no sqlite3.h file on the virtual machines. After installing libsqlite3-dev this file will be present, but the build will still fail. To solve the issue you need to add the required flag to bazel/external/system_libraries.BUILD:
cc_library(
    name = "libsqlite3-dev",
    linkopts = ["-lsqlite3"],
)
And then include it in the lte/gateway/c/core/BUILD.bazel at the end of MME_DEPS like this: "@system_libraries//:libsqlite3-dev",

Hey @vktng I made these changes to my PR (https://github.com/magma/magma/pull/13470/files#diff-5fef9f45d8fae0e6463b7bf908431d40936706fa860f0bb56002b3d00b77b285 and https://github.com/magma/magma/pull/13470/files#diff-5fe38d183eb821ecf919c6774f8b64769253b00d036d265dd853f95dcd7be484) but the BAZEL test still fails. Am I missing something else? Thanks!

Aug 03 '22 18:08 vroon2703

The Bazel build fails, because there is no sqlite3.h file on the virtual machines. After installing libsqlite3-dev this file will be present, but the build will still fail. To solve the issue you need to add the required flag to bazel/external/system_libraries.BUILD:
cc_library(
    name = "libsqlite3-dev",
    linkopts = ["-lsqlite3"],
)
And then include it in the lte/gateway/c/core/BUILD.bazel at the end of MME_DEPS like this: "@system_libraries//:libsqlite3-dev",
Hey @vktng I made these changes to my PR (https://github.com/magma/magma/pull/13470/files#diff-5fef9f45d8fae0e6463b7bf908431d40936706fa860f0bb56002b3d00b77b285 and https://github.com/magma/magma/pull/13470/files#diff-5fe38d183eb821ecf919c6774f8b64769253b00d036d265dd853f95dcd7be484) but the BAZEL test still fails. Am I missing something else? Thanks!

The dependency is still not on the VMs and the bazel-base container. You need to create a separate pull request where you add libsqlite3-dev as a dependency to them, merge it, and then this PR should pass. This should also resolve your issue with the failing mme test job. If you have more Bazel related questions you can find us in the #bazel Slack channel.

Aug 04 '22 07:08 vktng

@rdefosse, could you please help fix Magma-OAI-Jenkins CI failure here? I see from the logs that it is missing the libsqlite3-dev package on the container running in CI. Which docker files need to be updated for this CI job?

Aug 08 '22 19:08 ssanadhya

@ssanadhya @vroon2703

Sorry I did not see the notification.

Working on it.

Aug 09 '22 12:08 rdefosse

@vroon2703

There are 4 files you SHALL modify for this PR to pass on OAI pipeline.

I've upgraded the base images to include libsqlite3x and libsqlite3x-dev

diff --git a/ci-scripts/docker/Dockerfile.mme.ci.rhel8 b/ci-scripts/docker/Dockerfile.mme.ci.rhel8
index aee5fbbaad..7ab57caba8 100644
--- a/ci-scripts/docker/Dockerfile.mme.ci.rhel8
+++ b/ci-scripts/docker/Dockerfile.mme.ci.rhel8
@@ -65,6 +65,7 @@ RUN yum update -y && \
       libubsan \
       libasan \
       liblsan \
+      libsqlite3x \
       psmisc \
       procps-ng \
       tcpdump \
diff --git a/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18 b/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18
index 188519853d..21f7fe5935 100644
--- a/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18
+++ b/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18
@@ -70,6 +70,7 @@ RUN apt-get update && \
       net-tools \
       tshark \
       tzdata \
+      libsqlite3-0 \
   && rm -rf /var/lib/apt/lists/*
 
 # Copy pre-built shared object files
diff --git a/lte/gateway/docker/mme/Dockerfile.rhel8 b/lte/gateway/docker/mme/Dockerfile.rhel8
index 2d9f2f805b..88b96b560e 100644
--- a/lte/gateway/docker/mme/Dockerfile.rhel8
+++ b/lte/gateway/docker/mme/Dockerfile.rhel8
@@ -45,6 +45,8 @@ RUN rm /etc/rhsm-host && \
     protobuf-compiler \
     ruby \
     ruby-devel \
+    libsqlite3x \
+    libsqlite3x-devel \
   && ln -s /usr/bin/python3 /usr/bin/python \
   && ln -s /usr/bin/cmake3 /usr/bin/cmake
 
@@ -427,6 +429,7 @@ RUN yum update -y && \
       libubsan \
       libasan \
       liblsan \
+      libsqlite3x \
       psmisc \
       procps-ng \
       tcpdump \
diff --git a/lte/gateway/docker/mme/Dockerfile.ubuntu18.04 b/lte/gateway/docker/mme/Dockerfile.ubuntu18.04
index 23581cb0d9..596c2b2689 100644
--- a/lte/gateway/docker/mme/Dockerfile.ubuntu18.04
+++ b/lte/gateway/docker/mme/Dockerfile.ubuntu18.04
@@ -17,7 +17,7 @@ RUN [ "/bin/bash", "-c", "echo \"Install general purpose packages\" && \
     apt-get update && \
     DEBIAN_FRONTEND=noninteractive apt-get upgrade -y && \
     DEBIAN_FRONTEND=noninteractive apt-get install -y gnupg wget software-properties-common autoconf automake \
-    libtool curl make g++ unzip git build-essential autoconf libtool pkg-config \
+    libtool curl make g++ unzip git build-essential autoconf libtool pkg-config libsqlite3-dev libsqlite3-0 \
     gcc-7 g++-7 apt-transport-https ca-certificates apt-utils vim redis-server tzdata \
     libssl-dev ninja-build golang python2.7 automake perl libgmp3-dev clang-format-7 && \
     echo \"Configure C/C++ compiler v7.5 as primary\" && \
@@ -275,6 +275,7 @@ RUN apt-get update && \
       net-tools \
       tshark \
       tzdata \
+      libsqlite3-0 \
   && rm -rf /var/lib/apt/lists/*
 
 # Copy pre-built shared object files

Aug 09 '22 13:08 rdefosse

Thanks for the detailed response @rdefosse ! I have added these changes as a separate PR https://github.com/magma/magma/pull/13575 . Once that is merged, this PR should be green on CI. cc: @vroon2703

@vroon2703

There are 4 files you SHALL modify for this PR to pass on OAI pipeline.

I've upgraded the base images to include libsqlite3x and libsqlite3x-dev

diff --git a/ci-scripts/docker/Dockerfile.mme.ci.rhel8 b/ci-scripts/docker/Dockerfile.mme.ci.rhel8
index aee5fbbaad..7ab57caba8 100644
--- a/ci-scripts/docker/Dockerfile.mme.ci.rhel8
+++ b/ci-scripts/docker/Dockerfile.mme.ci.rhel8
@@ -65,6 +65,7 @@ RUN yum update -y && \
       libubsan \
       libasan \
       liblsan \
+      libsqlite3x \
       psmisc \
       procps-ng \
       tcpdump \
diff --git a/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18 b/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18
index 188519853d..21f7fe5935 100644
--- a/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18
+++ b/ci-scripts/docker/Dockerfile.mme.ci.ubuntu18
@@ -70,6 +70,7 @@ RUN apt-get update && \
       net-tools \
       tshark \
       tzdata \
+      libsqlite3-0 \
   && rm -rf /var/lib/apt/lists/*
 
 # Copy pre-built shared object files
diff --git a/lte/gateway/docker/mme/Dockerfile.rhel8 b/lte/gateway/docker/mme/Dockerfile.rhel8
index 2d9f2f805b..88b96b560e 100644
--- a/lte/gateway/docker/mme/Dockerfile.rhel8
+++ b/lte/gateway/docker/mme/Dockerfile.rhel8
@@ -45,6 +45,8 @@ RUN rm /etc/rhsm-host && \
     protobuf-compiler \
     ruby \
     ruby-devel \
+    libsqlite3x \
+    libsqlite3x-devel \
   && ln -s /usr/bin/python3 /usr/bin/python \
   && ln -s /usr/bin/cmake3 /usr/bin/cmake
 
@@ -427,6 +429,7 @@ RUN yum update -y && \
       libubsan \
       libasan \
       liblsan \
+      libsqlite3x \
       psmisc \
       procps-ng \
       tcpdump \
diff --git a/lte/gateway/docker/mme/Dockerfile.ubuntu18.04 b/lte/gateway/docker/mme/Dockerfile.ubuntu18.04
index 23581cb0d9..596c2b2689 100644
--- a/lte/gateway/docker/mme/Dockerfile.ubuntu18.04
+++ b/lte/gateway/docker/mme/Dockerfile.ubuntu18.04
@@ -17,7 +17,7 @@ RUN [ "/bin/bash", "-c", "echo \"Install general purpose packages\" && \
     apt-get update && \
     DEBIAN_FRONTEND=noninteractive apt-get upgrade -y && \
     DEBIAN_FRONTEND=noninteractive apt-get install -y gnupg wget software-properties-common autoconf automake \
-    libtool curl make g++ unzip git build-essential autoconf libtool pkg-config \
+    libtool curl make g++ unzip git build-essential autoconf libtool pkg-config libsqlite3-dev libsqlite3-0 \
     gcc-7 g++-7 apt-transport-https ca-certificates apt-utils vim redis-server tzdata \
     libssl-dev ninja-build golang python2.7 automake perl libgmp3-dev clang-format-7 && \
     echo \"Configure C/C++ compiler v7.5 as primary\" && \
@@ -275,6 +275,7 @@ RUN apt-get update && \
       net-tools \
       tshark \
       tzdata \
+      libsqlite3-0 \
   && rm -rf /var/lib/apt/lists/*
 
 # Copy pre-built shared object files

Aug 11 '22 04:08 ssanadhya

@vroon2703 , Changes for Magma-OAI-Jenkins are merged in https://github.com/magma/magma/pull/13575. Could you please rebase this PR on the latest master to re-trigger CI checks?

Aug 17 '22 18:08 ssanadhya

@ardzoht , @vktng , could you please review and confirm if the requested changes have been addressed on this PR?

Aug 22 '22 22:08 ssanadhya

@vroon2703 , you will need to rebase onto latest master to re-trigger the check-semantic-pr check.

Sep 01 '22 19:09 ssanadhya

Force merging this PR since it was blocked on semantic PR check for several days. The PR title looks semantically correct.

Sep 07 '22 21:09 ssanadhya