WALKOFF
WALKOFF copied to clipboard
nsacyber
Nginx isn't starting
current docker stack services, it's been like this since I tried 30 minutes ago
PS D:\dev\WALKOFF> docker stack services walkoff ID NAME MODE REPLICAS IMAGE PORTS
2tlevhxrh2gy walkoff_app_nmap replicated 0/0 127.0.0.1:5000/walkoff_app_nmap:1.0.0
42kbvqsr6k12 walkoff_app_ssh replicated 0/0 127.0.0.1:5000/walkoff_app_ssh:1.0.0
8hwek6bhek9m walkoff_resource_registry replicated 1/1 registry:2
*:5000->5000/tcp
94voa2ik6obx walkoff_core_worker replicated 0/0 127.0.0.1:5000/worker:latest
aeijazmi07bc walkoff_resource_nginx replicated 0/1 bitnami/nginx:latest *:8080->8080/tcp
b29hekf4ye1p walkoff_resource_portainer replicated 1/1 portainer/portainer:latest
dxs2sed9x4g8 walkoff_app_walk_off replicated 0/0 127.0.0.1:5000/walkoff_app_walk_off:1.0.0
fg9e4fwh588b walkoff_resource_redis replicated 1/1 redis:latest
*:6379->6379/tcp
ik51fiymgcmo walkoff_app_hive replicated 0/0 127.0.0.1:5000/walkoff_app_hive:1.0.0
k0qnoehvirhl walkoff_app_ip_addr_utils replicated 0/0 127.0.0.1:5000/walkoff_app_ip_addr_utils:1.0.0
morcj465ozoi walkoff_app_sdk replicated 0/0 127.0.0.1:5000/walkoff_app_sdk:latest
mzx14hl1o69k walkoff_resource_minio replicated 1/1 minio/minio:latest
olaf1aiuub54 walkoff_core_umpire replicated 0/1 127.0.0.1:5000/umpire:latest pdbkm6dg1fg8 walkoff_app_adversary_hunting replicated 0/0 127.0.0.1:5000/walkoff_app_adversary_hunting:1.0.0 rbxro4kdtarp walkoff_app_mitre_attack replicated 0/0 127.0.0.1:5000/walkoff_app_mitre_attack:1.0.0 snu26p7pj7uv walkoff_app_basics replicated 0/0 127.0.0.1:5000/walkoff_app_basics:1.0.0 t3kr4y37yyvl walkoff_app_power_shell replicated 0/0 127.0.0.1:5000/walkoff_app_power_shell:1.0.0 unxd9lyay47v walkoff_core_api_gateway replicated 0/1 127.0.0.1:5000/api_gateway:latest y4d2ch0fi1mf walkoff_resource_postgres replicated 1/1 postgres:latest
Walkoff appears to have been broken on Windows since beta 1 was released. Can this please be fixed so it can be stable tagged as working on Windows? I'm trying to build an app that uses Walkoff as the automation layer and even if I can manage to get it working temporarily, it's unshippable if there is no stable dependency that runs.
Does this still tie in with the last issue? We had changed the windows setup to mirror the linux setup so try using the walkoff.ps1 installation. The installation instructions need to be updated and the old windows_setup.ps1 need to be deleted apologies for the oversight. You should be able to use the bootloader using the same commands at the Linux version if you use walkoff.ps1. So for example walkoff.ps1 up --build and walkoff.ps1 down should work. Let us know if this solves the problem or its something else.
I'm not sure if this ties in to the last issue or not. walkoff.ps1 isn't working. This report is based on walkoff.ps1
Looking at the logs from https://github.com/nsacyber/WALKOFF/issues/239, I wonder if it's a symptom of the same issue.
The logs from #239 are due to running walkoff_setup.ps1 instead of walkoff.ps1 (there was an errant :5000 left over in the command retrieving the host's DockerNAT IP) - I'll push that deletion to master soon.
For this one, could you post the results of docker service ps service_name --no-trunc for the services that are at 0/1?
-
walkoff_resource_nginx -
walkoff_core_umpire -
walkoff_core_api_gateway
Additionally, are you using Docker for Windows (Hyper-V backend) or the older Docker Toolbox for Windows (Virtualbox backend)? @iadgovuser11 and I are using the former for testing on Windows. I can try and get the latter set up to try and replicate if that is the case.
I'm on the hyper v backend.
walkoff_resource_nginx docker service ps walkoff_resource_nginx --no-trunc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS ezxfwezamx9gzodwktx5c1rtz walkoff_resource_nginx.1 bitnami/nginx:latest@sha256:a0be1c01c8bd56d9609b612eb8052c1b1d1f6ec44ef904f666567f5644b6eb63 docker-desktop Ready Ready less than a second ago put0jh8dsrn08m969nhac182y _ walkoff_resource_nginx.1 bitnami/nginx:latest@sha256:a0be1c01c8bd56d9609b612eb8052c1b1d1f6ec44ef904f666567f5644b6eb63 docker-desktop Shutdown Failed less than a second ago "task: non-zero exit (1)" bo52v029cykqw7m7ivi9v3cgx _ walkoff_resource_nginx.1 bitnami/nginx:latest@sha256:a0be1c01c8bd56d9609b612eb8052c1b1d1f6ec44ef904f666567f5644b6eb63 docker-desktop Shutdown Failed 10 seconds ago "task: non-zero exit (1)" xmkyx09lypu9zeqweh8pzfjrg _ walkoff_resource_nginx.1 bitnami/nginx:latest@sha256:a0be1c01c8bd56d9609b612eb8052c1b1d1f6ec44ef904f666567f5644b6eb63 docker-desktop Shutdown Failed 19 seconds ago "task: non-zero exit (1)" v5a05wzb4k2t7kme97u82lvbz _ walkoff_resource_nginx.1 bitnami/nginx:latest@sha256:a0be1c01c8bd56d9609b612eb8052c1b1d1f6ec44ef904f666567f5644b6eb63 docker-desktop Shutdown Failed 28 seconds ago "task: non-zero exit (1)"
docker service ps walkoff_core_umpire --no-trunc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 61wf1nx59lxu54r49smrztlou walkoff_core_umpire.1 127.0.0.1:5000/umpire:latest docker-desktop Ready Rejected less than a second ago "No such image: 127.0.0.1:5000/umpire:latest" pec5rstjc0m2qiqsfmw5i0w20 _ walkoff_core_umpire.1 127.0.0.1:5000/umpire:latest docker-desktop Shutdown Rejected 5 seconds ago "No such image: 127.0.0.1:5000/umpire:latest" 3xyqh9ii3nhqhfii4mwkxc3vc _ walkoff_core_umpire.1 127.0.0.1:5000/umpire:latest docker-desktop Shutdown Rejected 10 seconds ago "No such image: 127.0.0.1:5000/umpire:latest" wnokul17b0egng3kmo3ez82j1 _ walkoff_core_umpire.1 127.0.0.1:5000/umpire:latest docker-desktop Shutdown Rejected 15 seconds ago "No such image: 127.0.0.1:5000/umpire:latest" kmuy9yeq5ky6wshg15u0fxr0n _ walkoff_core_umpire.1 127.0.0.1:5000/umpire:latest docker-desktop Shutdown Rejected 20 seconds ago "No such image: 127.0.0.1:5000/umpire:latest"
docker service ps walkoff_core_api_gateway --no-trunc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS x3v1jpdui1v4vunn8b1gf9b3h walkoff_core_api_gateway.1 127.0.0.1:5000/api_gateway:latest docker-desktop Ready Rejected 2 seconds ago "No such image: 127.0.0.1:5000/api_gateway:latest" 6bd0icaf3bo2chu45r6r4lews _ walkoff_core_api_gateway.1 127.0.0.1:5000/api_gateway:latest docker-desktop Shutdown Rejected 7 seconds ago "No such image: 127.0.0.1:5000/api_gateway:latest" fa92ylbubtrimcm36ktbslh1g _ walkoff_core_api_gateway.1 127.0.0.1:5000/api_gateway:latest docker-desktop Shutdown Rejected 12 seconds ago "No such image: 127.0.0.1:5000/api_gateway:latest" o4x61wrox95ru5uutj1dgz5cx _ walkoff_core_api_gateway.1 127.0.0.1:5000/api_gateway:latest docker-desktop Shutdown Rejected 17 seconds ago "No such image: 127.0.0.1:5000/api_gateway:latest" ox1im3tm26cjs2nhfibj5tejd _ walkoff_core_api_gateway.1 127.0.0.1:5000/api_gateway:latest docker-desktop Shutdown Rejected 22 seconds ago "No such image: 127.0.0.1:5000/api_gateway:latest"
Try walkoff.ps1 up --build it should re-download and build the images that are missing
Interestingly enough, walkoff.ps1 down seems to be rebuiliding everything and not working either. Seems like down isn't actually executing, and executes the same as up
Could you provide the output for what you're seeing? I'm unable to replicate that behavior on Windows.
Yeah, correction on the down one, that was my misreading it. Sorry, I'm out of it today.
Build didn't fix the problem though.
While the walkoff_resource_registry service is running, In PowerShell, can you run curl http://localhost:5000/v2/_catalog | Select-Object -ExpandProperty Content and send the output?
Additionally, the images that the docker service ps gave you, are those present when you do docker images?
I'm in the process of doing it. Also, I notice the following at the end of every time I start walkoff.ps1 with up.
koff_app_mitre_attack (id: mrf5mtjiiaet39pud0e21ql57) 2019-09-06 06:44:12,578 - BOOTLOADER - INFO:Updating service walkoff_core_worker (id: bj9ojho6jqgt1qojpbfgfaur0) 2019-09-06 06:44:12,578 - BOOTLOADER - INFO:Walkoff stack deployed, it may take a little time to converge. Use 'docker stack services walkoff' to check on Walkoff services. Web interface should be available at 'https://127.0.0.1:8080' once walkoff_resource_nginx is up. 2019-09-06 06:44:12,579 - UMPIRE - INFO:Docker connection closed.
could the docker connection closing cause part of this?
the docker connection closed is standard bootloader output. It just means the bootloader connection closed.
PS D:\dev\WALKOFF> docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5c9440f958bd portainer/portainer:latest "/portainer" 30 seconds ago Up 24 seconds 9000/tcp walkoff_resource_portainer.1.a5c7lkbwiva1ns80y5x87mvz1 50fdd5734b62 minio/minio:latest "/usr/bin/docker-ent…" 8 hours ago Up 8 hours 9000/tcp walkoff_resource_minio.1.fcx6bn3sbkf9pzvw4gbcurhrs 709b0e95aecb postgres:latest "docker-entrypoint.s…" 8 hours ago Up 8 hours 5432/tcp walkoff_resource_postgres.1.7z2sb45n6qnqzsw324ih56nqf 1de883ae9e36 registry:2 "/entrypoint.sh /etc…" 8 hours ago Up 8 hours 5000/tcp walkoff_resource_registry.1.ui6p0ixrjvkpy1zmwgep5cw2o 54a534a9a865 redis:latest "docker-entrypoint.s…" 8 hours ago Up 8 hours 6379/tcp walkoff_resource_redis.1.y90ai8uv04higx53h59laq6pp f8170beee17c prismagraphql/prisma:1.34 "/bin/sh -c /app/sta…" 8 days ago Up 8 hours 0.0.0.0:4466->4466/tcp nestjs-prisma-starter_prisma_1 20b702f49f19 mongo:3.6 "docker-entrypoint.s…" 8 days ago Up 8 hours 0.0.0.0:27017->27017/tcp nestjs-prisma-starter_mongo_1
PS D:\dev\WALKOFF> curl http://localhost:5000/v2/_catalog | Select-Object -ExpandProperty Content % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 37 100 37 0 0 37 0 0:00:01 --:--:-- 0:00:01 787 Select-Object : Property "Content" cannot be found. At line:1 char:42 + ... ://localhost:5000/v2/_catalog | Select-Object -ExpandProperty Content + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: ({"repositories":["walkoff_app_sdk"]}:PSObject) [Select-Object], PSArgumentException
- FullyQualifiedErrorId : ExpandPropertyNotFound,Microsoft.PowerShell.Commands.SelectObjectCommand
PS D:\dev\WALKOFF> docker images REPOSITORY TAG IMAGE ID CREATED SIZE bitnami/nginx
docker service ls ID NAME MODE REPLICAS IMAGE PORTS ky9w6sa8eyac walkoff_app_adversary_hunting replicated 0/0 127.0.0.1:5000/walkoff_app_adversary_hunting:1.0.0 zkrawkjeuz62 walkoff_app_basics replicated 0/0 127.0.0.1:5000/walkoff_app_basics:1.0.0 dueo9appdnvi walkoff_app_hive replicated 0/0 127.0.0.1:5000/walkoff_app_hive:1.0.0 vhb37qseg10n walkoff_app_ip_addr_utils replicated 0/0 127.0.0.1:5000/walkoff_app_ip_addr_utils:1.0.0 mrf5mtjiiaet walkoff_app_mitre_attack replicated 0/0 127.0.0.1:5000/walkoff_app_mitre_attack:1.0.0 ns8xyinwpwd8 walkoff_app_nmap replicated 0/0 127.0.0.1:5000/walkoff_app_nmap:1.0.0 zozmacbatqd1 walkoff_app_power_shell replicated 0/0 127.0.0.1:5000/walkoff_app_power_shell:1.0.0 3aze4359t22r walkoff_app_sdk replicated 0/0 127.0.0.1:5000/walkoff_app_sdk:latest sgw0z4jzm4vg walkoff_app_ssh replicated 0/0 127.0.0.1:5000/walkoff_app_ssh:1.0.0 p8qfpxgyolvn walkoff_app_walk_off replicated 0/0 127.0.0.1:5000/walkoff_app_walk_off:1.0.0 joee0ghaj6vk walkoff_core_api_gateway replicated 0/1 127.0.0.1:5000/api_gateway:latest p2qdh15kucgk walkoff_core_umpire replicated 0/1 127.0.0.1:5000/umpire:latest bj9ojho6jqgt walkoff_core_worker replicated 0/0 127.0.0.1:5000/worker:latest fqtxeto07kuf walkoff_resource_minio replicated 1/1 minio/minio:latest
j44ibx94wlw0 walkoff_resource_nginx replicated 0/1 bitnami/nginx:latest *:8080->8080/tcp
ysnonab0gyqx walkoff_resource_portainer replicated 1/1 portainer/portainer:latest
ln1y56vj43rp walkoff_resource_postgres replicated 1/1 postgres:latest
f5reo4n08gtn walkoff_resource_redis replicated 1/1 redis:latest *:6379->6379/tcp upprn8hblpjo walkoff_resource_registry replicated 1/1 registry:2 *:5000->5000/tcp
i can open localhost:5000 in a browser, the page is blank. no error though.
So it looks like it's failing somewhere in the build process - only walkoff_app_sdk is showing up. When you do walkoff.ps1 up --build --debug what output do you see?
Sorry, I'm kind of at a loss. Nothing in that build output suggests that it's having trouble building or pushing, but your docker images and curl output show that all of the images are missing except app_sdk. Was the behavior the same (docker service ps showing missing images) after that latest build?
Is there anything that might be cleaning up images on your system? If you manually build something and then push it to WALKOFF's registry while it's running, does it show up in docker images and the registry catalog? For example:
# Inside the WALKOFF directory
docker build -f api_gateway/Dockerfile -t 127.0.0.1:5000/api_gateway .
docker push 127.0.0.1:5000/api_gateway
.\walkoff.ps1 down --debug
.\walkoff.ps1 up --debug
So it's working now. I'm not sure if I should close the issue or not, but my gut tells me there's a deeper issue here.
So I just did a clean install and I had to do the manual build and push again. Shouldn't this be automated?
I just moved my development environment out of a Ubuntu VM and into Windows. Even on a clean install it still builds and pushes as expected in the walkoff.ps1 script, so I'm not really sure what the issue might be on your end. We're wrapping up some efforts for a release over the next few days, I'll get back to this once we release.
Alright, I'm going to look in to this on my end. A hack might be needed, if I can fix it on my end, I'll send a pull request.
Also, I have a theory that #245 might be related to this as well. If something is screwed up with the way it works on my machine (which is weird because docker usually works, but for some reason it stopped working right around the time the bootloader got implemented), it might be causing some kind of issue with how the push is handled. I don't know what would cause it though, as I'm running the latest docker for windows in a relatively vanilla windows 10.
Wait a second, just tested development branch and it seems to work out of the box.
#245 still seems to be a problem though. I'm wondering if it's related.
I'm going to close this issue for now. #245 seems to be having similar issues but for everything but bootloader and the api gateway. I have no idea why.
Wait, I think I was wrong.
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
walkoff_bootloader latest 7660c1f0526e About a minute ago 427MB
the api gateway is still the manually added one. Not the fresh one. Seems to not be pushing still.