origin icon indicating copy to clipboard operation
origin copied to clipboard

Long Execution Time for dotnet restore in OpenShift Build Pipeline

Open linhvuquach opened this issue 1 year ago • 6 comments

During our build pipeline execution on OpenShift, we've encountered a significant delay specifically with the dotnet restore command in the Dockerfile. This delay is causing notable slowdowns in our overall build process.

My Dockerfile

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
EXPOSE 8080
EXPOSE 8081

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
ARG HTTP_PROXY="<my-url-local-proxy>"
ENV DOTNET_NUGET_SIGNATURE_VERIFICATION=false
WORKDIR /src
COPY ["My.API/My.API.csproj", "My.API/"]
COPY ["My.Application/My.Application.csproj", "My.Application/"]
COPY ["My.Infrastructure/My.Infrastructure.csproj", "My.Infrastructure/"]
COPY "My.API/nuget.config" .

RUN dotnet restore -v diag "./My.API/./My.API.csproj"
COPY . .
WORKDIR "/src/My.API"
RUN dotnet build "./My.API.csproj" -c $BUILD_CONFIGURATION -o /app/build

FROM build AS publish
RUN dotnet publish "./My.API.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .

ENTRYPOINT ["dotnet", "My.API.dll"]

My nugget.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
</configuration>
Version

I'm using openshift version 4.

Current Result
  • Local result: I can work fine with my Dockerfile in my local.
  • OpenShift result: The dotnet restore operation consistently takes 1-hour duration and seems to time out during the MSBuild within our OpenShift build pipeline.
Additional Information
  • I saw a make sense information about NuGet signed-package verification https://github.com/NuGet/Home/issues/13062 and additional ENV DOTNET_NUGET_SIGNATURE_VERIFICATION=false in Dockerfile. But it didn't resolve yet.
    • You also need it for more information: https://learn.microsoft.com/en-us/dotnet/core/tools/nuget-signed-package-verification
  • In my Dockerfile I also add a Proxy variable and use nuget.config file. You can look at it above.
  • I tried to create another Node simple application image. But it also stuck on the npm install step. I think the same concept with the dotnet restore command.
  • I'm always looking at my resource quota. I don't see CPU, memory,... reach limit: image
  • I also tried to change aspnet:8.0 to aspnet:7.0. It still didn't success

Any insights or guidance will be expected. Thank you for your attention to this matter.

linhvuquach avatar Feb 23 '24 05:02 linhvuquach

Update

I'm confused about the requests % in the Observe function

CPU quota

cpuquota

Memory quota

memquota

linhvuquach avatar Feb 28 '24 10:02 linhvuquach

I had a same issue and my workaround is like that : added enviroment values (even for some of them are already used in the sdk):

ENV DOTNET_USE_POLLING_FILE_WATCHER=false 
ENV NUGET_XMLDOC_MODE=skip 
ENV NUGET_CERT_REVOCATION_MODE=offline 

use mount=type=cache in each restore / build / publish I have multiple services in one docker-compose and with that, I was able to decrease restore/build/publish time (bcs. of using same cache volume) time from like 1hour to 5min -- I'm using low personal NAS-server for it so time are higher as on normal Win-PC

Dockerfile for single service:

FROM mcr.microsoft.com/dotnet/aspnet:8.0.2-alpine3.19-composite-arm64v8 AS base
WORKDIR /app
RUN apk update && apk add curl
RUN rm -rf /var/cache/apk/*
HEALTHCHECK --interval=5s --timeout=10s --retries=5 --start-period=30s \
  CMD curl -f http://localhost:8080/api/health || exit 1
EXPOSE 8080

FROM mcr.microsoft.com/dotnet/sdk:8.0-alpine AS restorer
ARG BUILD_CONFIGURATION=Release
ARG BUILD_PLATFORM=linux-arm64
ENV DOTNET_NUGET_SIGNATURE_VERIFICATION=false
ENV DOTNET_USE_POLLING_FILE_WATCHER=false
ENV NUGET_XMLDOC_MODE=skip
ENV NUGET_CERT_REVOCATION_MODE=offline
RUN mkdir /src
WORKDIR /src
COPY ["Services/Catalog/Catalog.API/Catalog.API.csproj", "./Services/Catalog/Catalog.API/"]
COPY ["Services/Catalog/Catalog.Application/Catalog.Application.csproj", "./Services/Catalog/Catalog.Application/"]
COPY ["Services/Catalog/Catalog.Infrastructure/Catalog.Infrastructure.csproj", "./Services/Catalog/Catalog.Infrastructure/"]
COPY ["Services/Catalog/Catalog.Core/Catalog.Core.csproj", "./Services/Catalog/Catalog.Core/"]
COPY ["Services/Shared/Shared.Api/Shared.Api.csproj", "./Services/Shared/Shared.Api/"]
COPY ["Services/Shared/Shared.Application/Shared.Application.csproj", "./Services/Shared/Shared.Application/"]
COPY ["Services/Shared/Shared.Infrastructure/Shared.Infrastructure.csproj", "./Services/Shared/Shared.Infrastructure/"]
COPY ["Services/Shared/Shared.Core/Shared.Core.csproj", "./Services/Shared/Shared.Core/"]
WORKDIR "/src/Services/Catalog/Catalog.API"
RUN \
    --mount=type=cache,id=nugethttpcache,sharing=locked,target=/root/.local/share/NuGet/http-cache \
    --mount=type=cache,id=nugetglobalpackages,sharing=locked,target=/root/.nuget/packages/ \
    --mount=type=cache,id=nugettemp,sharing=locked,target=/Temp/NuGet \
    --mount=type=cache,id=nugetplugins-cache,sharing=locked,target=/root/.local/share/NuGet/plugin-cache \
#    dotnet restore "./Catalog.API.csproj" -r $BUILD_PLATFORM --disable-parallel
    dotnet restore "./Catalog.API.csproj" -r $BUILD_PLATFORM

FROM restorer AS build
WORKDIR /src
COPY . .
WORKDIR "/src/Services/Catalog/Catalog.API"
RUN \
    --mount=type=cache,id=nugethttpcache,sharing=locked,target=/root/.local/share/NuGet/http-cache \
    --mount=type=cache,id=nugetglobalpackages,sharing=locked,target=/root/.nuget/packages/ \
    --mount=type=cache,id=nugettemp,sharing=locked,target=/Temp/NuGet \
    --mount=type=cache,id=nugetplugins-cache,sharing=locked,target=/root/.local/share/NuGet/plugin-cache \
    dotnet build "./Catalog.API.csproj" -c $BUILD_CONFIGURATION -r $BUILD_PLATFORM --self-contained false -o /app/build 

FROM build AS publish
RUN \
    --mount=type=cache,id=nugethttpcache,sharing=locked,target=/root/.local/share/NuGet/http-cache \
    --mount=type=cache,id=nugetglobalpackages,sharing=locked,target=/root/.nuget/packages/ \
    --mount=type=cache,id=nugettemp,sharing=locked,target=/Temp/NuGet \
    --mount=type=cache,id=nugetplugins-cache,sharing=locked,target=/root/.local/share/NuGet/plugin-cache \
    dotnet publish "./Catalog.API.csproj" -c $BUILD_CONFIGURATION -r $BUILD_PLATFORM --self-contained false -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "/app/Catalog.API.dll"]

kopcot avatar Mar 14 '24 17:03 kopcot

This issue may occur when the build pod reaches its limit. Please try adding more resources to the Tekton builder pod, as shown below:

        - pipelineTaskName: build
          computeResources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1024Mi"
              cpu: "500m

Read more at: https://github.com/tektoncd/pipeline/blob/main/docs/pipelineruns.md#specifying-taskrunspecs

dungphamdev avatar Apr 05 '24 10:04 dungphamdev

It works for me if I add taskRunSpecs in the trigger-template.yaml . Thanks @dungphamdev

linhvuquach avatar Apr 09 '24 09:04 linhvuquach

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot avatar Jul 09 '24 01:07 openshift-bot

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot avatar Aug 08 '24 08:08 openshift-bot

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-bot avatar Sep 08 '24 00:09 openshift-bot

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci[bot] avatar Sep 08 '24 00:09 openshift-ci[bot]