IPinfo Sentinel Solution New Connectors

[AhmadMujahid2k]

Required items, please complete

Change(s):

• Added 14 new data connectors that use logs ingestion API to ingest data into custom tables.

Reason for Change(s):

• New data sets required.

Version Updated:

• Yes
• 3.0.1

Testing Completed:

• Yes

Checked that the validations are passing and have addressed any issues that are present:

• Need Help to resolve this Fail test case of arm-ttk : [-] IDs Should Be Derived From ResourceIDs (125 ms) Property: "id" must use one of the following expressions for an resourceId property: extensionResourceId,resourceId,subscriptionResourceId,tenantResourceId,if,parameters,reference,variables,subscription,guid
• Same Issue face in my previous PR #10553

[AhmadMujahid2k]

These are the invocation logs:

WHOIS NET.csv WHOIS POC.csv WHOIS ORG.csv WHOIS MNT.csv WHOIS ASN.csv RWHOIS.csv RIRWHOIS.csv Iplocation Extended.csv Domain.csv Privacy Extended.csv Abuse.csv ASN.csv Carrier.csv

[v-prasadboke]

Hello @AhmadMujahid2k, Thanks for raising this PR. This PR will be investigated and we will update you about the same before 29 August, 2024

[AhmadMujahid2k]

Invocation log for new Country+ASN data connector. Country_ASN.csv

[max-ipinfo]

@v-prasadboke could you provide an update whether you need any changes from us (IPinfo)? I will be continuing @AhmadMujahid2k's work moving forward.

Thank you!

[v-prasadboke]

@v-prasadboke could you provide an update whether you need any changes from us (IPinfo)? I will be continuing @AhmadMujahid2k's work moving forward.

Thank you!

Hello @AhmadMujahid2k & @max-ipinfo sorry for the late response. Had some priority tasks on my name. Right now we are upgrading the Python version of Function apps to 3.11.

We recommend you to upgrade the python version to 3.11 as 3.8 is deprecated and 3.9 & 3.10 will be deprecated soon

[max-ipinfo]

I updated all references to Python to version 3.11: https://github.com/Azure/Azure-Sentinel/pull/10981/commits/4ab1e11b0f625c20a0142dfd815f5c1159a750c0

@v-prasadboke are there any other remaining steps you would like me to perform?

[max-ipinfo]

@v-prasadboke could you provide an update? We are blocked on this PR to release our Solution offering to our customers.

Thank you.

[v-prasadboke]

Hello @AhmadMujahid2k

Getting following error

[max-ipinfo]

@v-prasadboke regarding your error, I was trying to address your comment asking us to upgrade to Python 3.11. All I did in https://github.com/Azure/Azure-Sentinel/pull/10981/commits/4ab1e11b0f625c20a0142dfd815f5c1159a750c0 is changing "linuxFxVersion" from "Python|3.10" to "Python|3.11".

I am unfortunately not up-to-speed with how to test a Sentinel Solution:

1. can you share how I can access the web interface you showed in these two last screenshots?
2. is there developer documentation I could follow to double-check the validation of our work?
3. do I need to regenerate zip files under Solutions/IPinfo? If so, how?

Thanks!

[max-ipinfo]

I went through the docs: https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#valid-linuxfxversion-values

I see that Python|3.11 is indeed supported:

$ az functionapp list-runtimes --os linux --query "[].{stack:join(' ', [runtime, version]), LinuxFxVersion:linux_fx_version, SupportedFunctionsVersions:to_string(supported_functions_versions[])}" --output table
Stack              LinuxFxVersion       SupportedFunctionsVersions
-----------------  -------------------  ----------------------------
dotnet-isolated 8  DOTNET-ISOLATED|8.0  ["4"]
dotnet-isolated 7  DOTNET-ISOLATED|7.0  ["4"]
dotnet-isolated 6  DOTNET-ISOLATED|6.0  ["4"]
dotnet 8           DOTNET|8.0           ["4"]
dotnet 6           DOTNET|6.0           ["4"]
node 20            Node|20              ["4"]
node 18            Node|18              ["4"]
python 3.11        Python|3.11          ["4"]
python 3.10        Python|3.10          ["4"]
python 3.9         Python|3.9           ["4"]
python 3.8         Python|3.8           ["4"]
python 3.7         Python|3.7           ["4"]
java 21.0          Java|21              ["4"]
java 17.0          Java|17              ["4"]
java 11.0          Java|11              ["4"]
java 8.0           Java|8               ["4"]
powershell 7.4     PowerShell|7.4       ["4"]
powershell 7.2     PowerShell|7.2       ["4"]
custom                                  ["4"]

$ az functionapp list-runtimes --os linux --query "[].{stack:join(' ', [runtime, version]), LinuxFxVersion:linux_fx_version, SupportedFunctionsVersions:to_string(supported_functions_versions[])}" --output table | grep 3.11
python 3.11        Python|3.11          ["4"]

That's the version set everywhere as far as I can tell:

ipinfo Solutions/IPinfo (IpinfoIntegration)$ rg linuxFxVersion
Data Connectors/Abuse/azuredeploy_Connector_IPinfo_Abuse_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/RWHOIS/azuredeploy_Connector_IPinfo_RWHOIS_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/WHOIS MNT/azuredeploy_Connector_IPinfo_WHOIS_MNT_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/WHOIS ORG/azuredeploy_Connector_IPinfo_WHOIS_ORG_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Country ASN/azuredeploy_Connector_IPinfo_Country_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Privacy Extended/azuredeploy_Connector_IPinfo_Privacy_Extended_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Company/azuredeploy_Connector_IPinfo_Company_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/WHOIS NET/azuredeploy_Connector_IPinfo_WHOIS_NET_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Privacy/azuredeploy_Connector_IPinfo_Privacy_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/WHOIS ASN/azuredeploy_Connector_IPinfo_WHOIS_ASN_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/WHOIS POC/azuredeploy_Connector_IPinfo_WHOIS_POC_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/RIRWHOIS/azuredeploy_Connector_IPinfo_RIRWHOIS_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Iplocation Extended/azuredeploy_Connector_IPinfo_Iplocation_Extended_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Carrier/azuredeploy_Connector_IPinfo_Carrier_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Iplocation/azuredeploy_Connector_IPinfo_Iplocation_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/Domain/azuredeploy_Connector_IPinfo_Domain_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

Data Connectors/ASN/azuredeploy_Connector_IPinfo_ASN_AzureFunction.json
117:                    "linuxFxVersion": "Python|3.11"

[v-prasadboke]

@v-prasadboke regarding your error, I was trying to address your comment asking us to upgrade to Python 3.11. All I did in 4ab1e11 is changing "linuxFxVersion" from "Python|3.10" to "Python|3.11".

I am unfortunately not up-to-speed with how to test a Sentinel Solution:

1. can you share how I can access the web interface you showed in these two last screenshots?

• Go to Azure portal.
• Select deploy a custom template
• paste the azure deploy file in the edit template section. But before this in the azure deploy file search for "website run from package" and change its value to zip of your repo
• after deploying head towards function page and select your function app

2. is there developer documentation I could follow to double-check the validation of our work?
3. do I need to regenerate zip files under Solutions/IPinfo? If so, how?

• You do not need to regenerate the zip just include all the changes in the zip

Thanks!

[v-prasadboke]

Hello @AhmadMujahid2k, Please provide your update on the above

[v-prasadboke]

Hello @AhmadMujahid2k, We are waiting for your feedback

[abdullahdevrel]

@max-ipinfo is leading this PR. Max, please update @v-prasadboke on the status of the PR, please.

[v-prasadboke]

@max-ipinfo, If possible can you provide us some update on this

[v-prasadboke]

Hello @AhmadMujahid2k & @max-ipinfo can you provide us some update on this

[v-prasadboke]

We wanted to check on the status of PR #10981 . PR is pending for more than 45+ days. Please let us know if you need any assistance to review this PR. Per our standard operating procedures if no response is received in the next 7 business days, we will close this PR. Thank you for your cooperation.

[AhmadMujahid2k]

Hi @Max and @Abdullah, I hope you both are doing well. Could you please share the required updates or respond to the reviewer to keep things moving? Thank you!

[max-ipinfo]

@v-prasadboke thank you for your message. We would definitely need assistance getting this PR moving again.

Last time I tried, I wasn't able to easily follow your instructions to debug the Python version error you were getting. So we've stuck unable to move it forward.

Any support on your side would be helpful @v-prasadboke . Thank you!

[max-ipinfo]

@v-prasadboke I just wanted to let you know that the IPinfo team had a deep-dive call with the Microsoft Sentinel team on Friday December 6, and it gave us great insight into the Sentinel Platform.

I am hoping to make progress on the PR this week. Thank you for your patience!

[v-prasadboke]

Thanks for the update @max-ipinfo

[v-prasadboke]

Hello @max-ipinfo, Do we have any update here

[v-prasadboke]

Hello @AhmadMujahid2k

Can you provide us an update or probably any ETA

[max-ipinfo]

Hello @v-prasadboke, sorry for my late reply. I was OOO for the holidays.

I am still working on testing our Sentinel Solution in Azure. Fortunately, I will have some time to continue my testing in the upcoming days and expect to provide a concrete update this week.

Thank you for your patience.

[max-ipinfo]

@v-prasadboke I was able to test the deployment of one Data Connector Azure Function to Azure with Python 3.11 without any problem.

Here is what I did:

az group create \
  --name $resourceGroupName \
  --location $location

az storage account create \
  --name $storageAccountName \
  --location $location \
  --resource-group $resourceGroupName \
  --sku Standard_LRS

az functionapp create \
    --resource-group $resourceGroupName \
    --consumption-plan-location $location \
    --os Linux \
    --runtime python \
    --runtime-version 3.11 \
    --functions-version 4 \
    --name $functionAppName \
    --storage-account $storageAccountName

az functionapp deployment source config-zip \
    --resource-group $resourceGroupName \
    --name $functionAppName \
    --src $zipFilePath

Here is the proof from my end:

Can you elaborate what are the remaining issues you would need us to handle?

[v-prasadboke]

Hey thanks for the update @max-ipinfo, i'll take a look at this. Also can you share invocation logs for the same.

Head to functions tab, click on function and you will have option for the logs. Please share a screenshot of the same.

Thanks, Prasad

[max-ipinfo]

Also can you share invocation logs for the same.

Head to functions tab, click on function and you will have option for the logs. Please share a screenshot of the same.

The logs are basically empty:

Any manual call that I make to the Azure Function (following the instructions on https://learn.microsoft.com/en-us/azure/azure-functions/functions-manually-run-non-http) do not do anything or show any logs or invocations:

functionKey=$(az functionapp keys list --resource-group $resourceGroupName --name $functionAppName | jq -r '.masterKey')
functionUrl=$(az functionapp function show --resource-group $resourceGroupName --name $functionAppName --function-name $functionName | jq -r '.href')

 curl --verbose -X POST $functionUrl -H "Content-Type: application/json"   -H "x-functions-key: ${functionKey}"  -d "{}"

I have spent the whole day trying to debug this without making progress.

How do you trigger the Azure Function in your test environment? How do you connect your Sentinel workspace fo these manually created Azure Functions used for Data Connectors?

[v-prasadboke]

Hello @AhmadMujahid2k, Can we connect on teams. my id is : [email protected]

Or we can set up a meet for the same

[max-ipinfo]

Hello @AhmadMujahid2k, Can we connect on teams. my id is : [email protected]

Or we can set up a meet for the same

As a reminder, I am finishing the PR.

Thanks for the contact info. I will reach out to you directly on Teams.

[max-ipinfo]

@v-prasadboke as you requested, I applied the required changes:

• host.json: replace Microsoft.Azure.Functions.ExtensionBundle' version "[4.*, 5.0.0)" " to "[3.*, 4.0.0)"
• Replace Azure function's linuxFxVersion from "Python|3.11" to "python|3.11"