office365-audit-log-collector icon indicating copy to clipboard operation
office365-audit-log-collector copied to clipboard
verified publisher icon ddbnl

error in latest version

Open vvhor opened this issue 1 year ago • 11 comments

Hello,

I'm trying the latest version but I got this error

thread 'main' panicked at src\api_connection.rs:59:33:
Could not parse API login reply: error decoding response body: missing field `access_token` at line 1 column 623
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

the request is:

./OfficeAuditLogCollector.exe --tenant-id "xxxxxxx" --client-id "xxxxx" --secret-key "xxxxx" --config config.yaml

config file:

collect:
  skipKnownLogs: True
  workingDir: ./
  maxThreads: 50
  globalTimeout: 5
  retries: 3
  hoursToCollect: 168
  contentTypes:
    Audit.General: True
    Audit.AzureActiveDirectory: True
    Audit.Exchange: True
    Audit.SharePoint: True
    DLP.All: True 
output:
  file:
    path: 'output.csv'
    separateByContentType: True
    separator: ';'

I'm using the client on window system

vvhor avatar Mar 21 '24 22:03 vvhor

Hi @vvhor,

Have you successfully used the older version(s) with the same app registration before, or are you trying for the first time? If it's the first time you could check if the API permissions are properly set, and if auditing is enabled for the tenant (this might take a while to sync after enabling it). Both these actions are described in README.md.

If it was already working before then we'll have to figure where it's coming from. I'm currently working on a new release with improved logging, so once that's out in the coming days I will link it here. Then hopefully we can see more with the increased logs.

ddbnl avatar Mar 22 '24 08:03 ddbnl

Hi,

I've used it in previous version on other tenant. With this tenant it's the first time.

I've followed all of the steps in the READEME some days ago

vvhor avatar Mar 22 '24 10:03 vvhor

I'm currently working on a new release with improved logging, so once that's out in the coming days I will link it here. Then hopefully we can see more with the increased logs.

did you have an estimation for this release?

vvhor avatar Mar 22 '24 11:03 vvhor

I have released the new version with fixed logging and also extended logging, hopefully we'll be able to capture the error:

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.1

Make sure to also enable logging in the config:

log:
  path: './log.txt'
  debug: True

If you get it working consider disabling debug again, it's very noisy. Let me know what it does for you.

ddbnl avatar Mar 22 '24 16:03 ddbnl

Hi,

many thanks. Now the log is very helpful. I'll do some test and let you know

vvhor avatar Mar 22 '24 17:03 vvhor

As a heads up, there's a new release that added an interactice interface that can be used for testing. If you have the new release, you can run the command as you did before, but add the '--interactive' command line parameter. This allows you to test the connection and immediately see the logs for any errors.

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.2

Screenshot

ddbnl avatar Mar 24 '24 23:03 ddbnl

Hello,

I'm now having different errors in "Run Collector":

[00:00:01.339] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.339] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.339] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.339] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.341] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.341] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.349] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.349] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.350] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.350] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.350] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.350] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.352] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.352] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.352] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.352] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.356] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.356] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.358] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.358] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.401] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.401] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.403] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.403] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.407] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.407] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.408] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.408] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.409] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.409] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.412] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.412] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.412] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.412] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.412] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.412] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.414] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.414] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.416] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.416] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.416] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.416] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.417] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.417] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.418] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.418] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.420] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.420] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.420] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.420] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.420] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.420] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.422] (52b4) INFO   Blobs found: 0
Blobs successful: 0
Blobs failed: 0
Blobs retried: 34
Logs saved: 0

[00:00:01.422] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.424] (2e58) ERROR  Err getting blob response error sending request for url (https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123): connection error: Either the application has not called WSAStartup, or WSAStartup failed. (os error 10093)
[00:00:01.424] (2e58) ERROR  Could not resend failed blob, dropping it: send failed because receiver is gone

Non error in "Test API Connection"

vvhor avatar Mar 25 '24 14:03 vvhor

That's odd, so far I'm not able to reproduce. Best we can do is improve logging. I've added the full output of the JSON response as a debug log, in the section where you are receiving the error. This should give us the full response you are getting from the API. Can you run it again with the latest release, and enabling debug logging?

Also, just to ensure you are not being rate limited, could you use a publisher ID? For the ID you can just use your tenant ID again. This will isolate your requests to avoid rate limiting as much as possible. You can the executable like before, but adding "--publisher-id %tenant-id%"

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.3

ddbnl avatar Mar 25 '24 22:03 ddbnl

PS. The new version not write the log in interactive mode.

in attach the full log log.txt

vvhor avatar Mar 27 '24 10:03 vvhor

Hi,

just to ask if you have any news about the errors..

vvhor avatar Mar 28 '24 14:03 vvhor

can I do any other test to help you?

vvhor avatar Apr 04 '24 12:04 vvhor