MeshCentral icon indicating copy to clipboard operation
MeshCentral copied to clipboard

AMT activated in ACM. But CIRA: Disconnected

Open ZENAdmin-Ops opened this issue 1 year ago • 48 comments

Describe the bug Hello,

I'm raising this as a bug as I have this issue on 4 devices and have not been able to determine a resolution. And the discussion questions that I have seen have not been responded to.

This could be a bug, or maybe I just need some troubleshooting advice.

I have other AMT devices (and agents) that are Activated and are connecting fine to the MC server. So the MC server is functional - at least to some degree.

To Reproduce Steps to reproduce the behavior:

  1. Have an Intel AMT only group
  2. Intel AMT Policy
  3. Simple ACM
  4. Keep existing password
  5. Deactivate CCM
  6. CIRA - Connect to server
  7. Deploy setup file using meshcmd
  8. AMT device appears in group, but IntelAMT tab is diconnected with no option to connect
  9. Run meshcmd amtinfo on device.
  10. Intel AMT v12.0.93, activated in Admin Control Mode (ACM). Wired Enabled, Static, E4:54:E8:92:A7:FB, 10.254.242.100 Connection Status: Direct, CIRA: Disconnected.

Expected behavior Expect CIRA to be connected

Screenshots If applicable, add screenshots to help explain your problem.

Server Software (please complete the following information):

  • OS: Windows 10 Pro
  • Virtualization: Hyper-V VM
  • Network: WANonly
  • Version: 1.1.24
  • Node: 20.15.0

Client Device (please complete the following information):

  • Device: OptiPlex 7070
  • OS: Windows 10 Pro
  • Network: [Remote over WAN]
  • Browser: [N/A]
  • MeshCentralRouter Version: [if applicable]

Remote Device (please complete the following information):

  • Device: [e.g. Laptop]
  • OS: [e.g. Windows 10 21H2]
  • Network: [e.g. Local to Meshcentral, Remote over WAN]
  • Current Core Version (if known): [HINT: Go to a device then console Tab then type info]

Additional context Add any other context about the problem here.

Your config.json file

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "cert": "mc1.zen.net.au",
    "WANonly": true,
	"_LANonly": true,
    "sessionKey": "### have changed ###",
    "_port": 443,
    "aliasPort": 5443,
	"MpsAliasPort": 54433,
    "_redirPort": 80,
    "_redirAliasPort": 80,	
	"mongodb": "mongodb://127.0.0.1:27017/meshcentral",
    "mongodbcol": "meshcentral",
	"autoBackup": {
	"mongoDumpPath": "C:/Program Files/MongoDB/Server/7.0/bin/mongodump.exe",
	"backupIntervalHours": 24,
	"keepLastDaysBackup": 10,
	"zipPassword": "#### have changed ####",
	"backupPath": "C:/MeshCentral/meshcentral-backups",
	"maxFiles": 10
	}
  },
  "domains": {
    "": {
      "_title": "MyServer",
      "_title2": "Servername",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "[email protected]",
    "names": "mc1.zen.net.au",
    "skipChallengeVerification": true,
    "production": false
  }
}

ZENAdmin-Ops avatar Jul 25 '24 06:07 ZENAdmin-Ops

you say Keep existing password, in the General tab of the device, is the an option saying invalid credentials next to the AMT bit at all? share screenshot of you can?

si458 avatar Jul 25 '24 08:07 si458

Some screenshots from MeshCentral Details_tab_not_connected General_tab_not_connected IntelAMT_tab_not_connected

ZENAdmin-Ops avatar Jul 25 '24 08:07 ZENAdmin-Ops

people keep reporting this type of issue and sadly i havent been able to replicate the issue? BUT i only have a single AMT v7 device which works no matter what, so without hardware to test, i cannot fix the bug (if it is a bug).

a few things you can try:

  1. go into the console tab under My Server then type amtpasswords then check the device thats listed in there AND the password for the AMT (mebx) is infact in that list.

  2. check you can access the AMT web panel on the device itself https://localhost:16992 or https://localhost:16993

  3. if you can access the web ui, check you can login with a password listed in amtpasswords for that device

  4. run meshcentral with amt debug node node_modules/meshcentral --debug amt

si458 avatar Jul 25 '24 08:07 si458

Hello,

Have I done the amtpasswords step correctly? Console_amtpasswords

ZENAdmin-Ops avatar Jul 25 '24 08:07 ZENAdmin-Ops

@ZENAdmin-Ops erm yes and im not sure why its blank?

si458 avatar Jul 25 '24 08:07 si458

Nothing else has appeared in the console

ZENAdmin-Ops avatar Jul 25 '24 08:07 ZENAdmin-Ops

@ZENAdmin-Ops can you just try all the other options for me tho please

si458 avatar Jul 25 '24 08:07 si458

I am able to logon to the device using https://localhost:16993

But there are no passwords listed in amtpasswords so I am using the credentials that I know are correct for the device

ZENAdmin-Ops avatar Jul 25 '24 08:07 ZENAdmin-Ops

For step 4.

Where will the logs / diagnostic info be reported that I will need to submit?

ZENAdmin-Ops avatar Jul 25 '24 09:07 ZENAdmin-Ops

@ZENAdmin-Ops sorry first stop meshcentral, then rerun it with that command, the logs will show on the screen

you can also go into the web ui, then my server, then trace, click tracing, then enable ALL the intel amt sections, then it will show any debug stuff in the web ui (dont close the page!)

if you use the web ui, its best to shut down computer, unplug its power, wait say 2 mins, then plug its power back in, switch computer on, then watch the web ui output

if you get no output in either the console or the trace ui, then you might have the amt port 4433 blocked in your firewall

si458 avatar Jul 25 '24 09:07 si458

OK. These systems are going to be in use this evening.

So I will try the tracing tomorrow and report back then.

Ironically I had access to AMT via VNC Plus prior to trying to switch them to MeshCentral. And at the moment AMT access via MeshCentral for these devices is broken.

So, I can't do the remote restart and wait 2 mins.

I'll just try a remote restart tomorrow. Worst case I'll have to go to the office to perform a power-drain, that will be sometime in the next week.

I'd really like to get to the bottom of this. So I'll do whatever testing is needed.

Appreciate your prompt assistance.

ZENAdmin-Ops avatar Jul 25 '24 09:07 ZENAdmin-Ops

thats ok! in theory you can do the web ui trace now if you wanted while its running,

go into the web ui, then my server, then trace, click tracing, then enable ALL the intel amt sections, then it will show any debug stuff in the web ui (dont close the page!)

then watch the logs as you try different things and see what happens,

but what you need to do is really unplug/plug the remote device, that will restart AMT into connecting to your meshcentral, and if you NEVER see any connections from that device happen when you power cycle it, then something isnt right with the AMT

strange one (side note), can you use the meshcommander application and connect into the device? if you can then you can check the internet settings tab and see if your meshcentral server dns is listed there, same with the security settings, check its certificate is listed there

si458 avatar Jul 25 '24 09:07 si458

The server trace doesn't appear to identify the host.

And I have AMT devices that are working as well as 3 or 4 that are stuffed. Server_Trace

ZENAdmin-Ops avatar Jul 25 '24 09:07 ZENAdmin-Ops

Internet Settings looks correct

I'm using a redirected port.

Can you tell me where I check the Trusted Root certificate? MeshCommader_Internet_Settings

ZENAdmin-Ops avatar Jul 25 '24 09:07 ZENAdmin-Ops

plz can you share a few more screenshots? just trying to compare with mine System Status, Network Settings, Internet Settings, Security Settings ?

si458 avatar Jul 25 '24 10:07 si458

ive just spotted, it says talk to port 54433, which you have set as your alias port for amt do you use a reverse proxy server at all? as the default port should be 4433, and the alias port only tells remote device which port to connect to, but you havent specified an amt port so its going to use the default 4433 port so 54433 != 4433

si458 avatar Jul 25 '24 10:07 si458

plz can you share a few more screenshots? just trying to compare with mine System Status, Network Settings, Internet Settings, Security Settings ?

MeshCommander_Network_Settings MeshCommander_System_Status MeshCommander_Security_Settings

ZENAdmin-Ops avatar Jul 25 '24 10:07 ZENAdmin-Ops

the dns suffix for the computer in incorrect! from my understanding and knowledge, the dns suffix MUST match your meshcentral dns name!? so you should be able to click the name & domain in network settings, and change the dns suffix

si458 avatar Jul 25 '24 10:07 si458

ive just spotted, it says talk to port 54433, which you have set as your alias port for amt do you use a reverse proxy server at all?

No reverse proxy server.

as the default port should be 4433, and the alias port only tells remote device which port to connect to, but you havent specified an amt port so its going to use the default 4433 port so 54433 != 4433

I don't understand the comment above.

The public AMT port is 54433 which forwards to 4433 on the MeshCentral server.

ZENAdmin-Ops avatar Jul 25 '24 10:07 ZENAdmin-Ops

The public AMT port is 54433 which forwards to 4433 on the MeshCentral server.

ah right ok thats fine! so port forwarding 54433 to 4433, thats ok makes sense! my mistake!

si458 avatar Jul 25 '24 10:07 si458

the dns suffix for the computer in incorrect! from my understanding and knowledge, the dns suffix MUST match your meshcentral dns name!? so you should be able to click the name & domain in network settings, and change the dns suffix

Well I have devices at other client sites that working in MeshCentral and their domain suffix does not match our dns suffix

If all the devices have to match MeshCentral dns suffix that will be a showstopper for us. We will need a work-around

Does this mean that I would need to set up multiple domains within MeshCentral. There is a video on multiple domains, but I haven't watched it yet.

ZENAdmin-Ops avatar Jul 25 '24 10:07 ZENAdmin-Ops

hmm ok that is weird? and forgive me but i could be mistaken!

in theory you can click the name & domain, then change name sharing to dedicated, different from os, then set a computer name and your meshcentral dns name in there (computer1.meshcentral.myserver.com) and then click OK/Save, it then WONT effect your windows OS domain as its seperate :)

si458 avatar Jul 25 '24 10:07 si458

hmm ok that is weird? and forgive me but i could be mistaken!

in theory you can click the name & domain, then change name sharing to dedicated, different from os, then set a computer name and your meshcentral dns name in there (computer1.meshcentral.myserver.com) and then click OK/Save, it then WONT effect your windows OS domain as its seperate :)

So you would suggest that the name should be: bbox4.zen.net.au ?

ZENAdmin-Ops avatar Jul 25 '24 10:07 ZENAdmin-Ops

So you would suggest that the name should be: bbox4.zen.net.au ?

no it would need to match the DNS name you use to access meshcentral, so from the screenshots/config.json something like comp1.mc1.zen.net.au

si458 avatar Jul 25 '24 10:07 si458

I've just changed the name to:

bbox4.mc1.zen.net.au

No change in amtinfo CIRA: Disconnected

Do I need to restart?

ZENAdmin-Ops avatar Jul 25 '24 10:07 ZENAdmin-Ops

@ZENAdmin-Ops possibly the remote machine yes, by unplugging/plugging back in, but maybe just give it chance to connect, as i do know/have seen my AMT can take 5-10mins before it connects back

si458 avatar Jul 25 '24 10:07 si458

@ZENAdmin-Ops what is the full output now if you run the amtinfo with meshcmd?

si458 avatar Jul 25 '24 10:07 si458

@ZENAdmin-Ops what is the full output now if you run the amtinfo with meshcmd?

I have restarted

C:\ZEN\MeshCentral\ZEN>meshcmd amtinfo Intel AMT v12.0.93, activated in Admin Control Mode (ACM). Wired Enabled, Static, E4:54:E8:92:A7:FB, 10.254.242.100 Connection Status: Direct, CIRA: Disconnected.

ZENAdmin-Ops avatar Jul 25 '24 10:07 ZENAdmin-Ops

ok i am confused because its missing a section which should show the dns suffix? but at the same time the amtinfo command has NEVER worked on my V7 due to a bug i will try fix my bug here next week, and then have a look at whats going on if i can im sorry i cant be of much more help, i will ask @Ylianst when i get chance, see if he has any suggestions as we must be missing something!

si458 avatar Jul 25 '24 10:07 si458

This is from a system that is working.

Later version of AMT.

C:\ZEN\MeshCentral>meshcmd amtinfo Intel AMT v16.1.30, activated in Admin Control Mode (ACM). Wired Enabled, DHCP, CC:96:E5:37:4A:7F DNS suffix: alleanza.local Connection Status: Outside, CIRA: Connected to mc1.zen.net.au, Periodic.

What determines Connection Status: Outside versus Direct?

ZENAdmin-Ops avatar Jul 25 '24 10:07 ZENAdmin-Ops