monero-project
CCS Wallet Incident
The CCS Wallet was drained of 2,675.73 XMR (the entire balance) on September 1, 2023, just before midnight. A second, hot wallet, used for payments to contributors, is untouched; its balance is ~244 XMR. We have thus far not been able to ascertain the source of the breach.
Timeline
- April 12, 2020: New CCS wallet is created by fluffypony (on a dedicated wallet laptop, a Purism Librem 14, running Qubes) and the seed shared with Luigi, half via the Wire app, and half via GPG-encrypted email -- fluffypony and Luigi are the only parties with known access to the CCS seed.
- 2020-2023: (Luigi's side) a single use Ubuntu system is set up to run a Monero node and CCS wallet; the hot wallet is on a Windows 10 Pro desktop where it has been since 2017; Luigi makes payments from the hot wallet and tops it up from the CCS Wallet (via SSH), occasionally as needed.
- August 3, 2021: shortly after fluffypony's arrest, most of the CCS wallet was swept by Luigi to the hot wallet as a short-term measure pending more information about the nature of the arrest
- (a few weeks/months later) fluffypony's arrest is determined not crypto-related; reverted to previous behavior of large CCS balance, small hot wallet balance
- May 10, 2023: last transfer was made by Luigi from CCS wallet to hot wallet
- September 1 11:58pm - September 2 12:07am, 2023: CCS wallet was swept in 9 transactions, IDs: ffc82e64dde43d3939354ca1445d41278aef0b80a7d16d7ca12ab9a88f5bc56a 08487d5dbf53dfb60008f6783d2784bc4c3b33e1a7db43356a0f61fb27ab90cc 4b73bd9731f6e188c6fcebed91cc1eb25d2a96d183037c3e4b46e83dbf1868a9 8a5ed5483b5746bd0fa0bc4b7c4605dda1a3643e8bb9144c3f37eb13d46c1441 56dd063f42775600adf03ae1e7d7376813d9640c65f08916e3802dbfee489e2c e2ab762927637fe0255246f8795a02bd7bb99f905ae7afc21284e6ff9e7f73db 9bf312ed09da1e7dfce281a76ae2fc5b7b9edc35d31c9eb46b21d38500716b6b 837de977651136c18b0018269626be7155d477cc731c5ca907608a2db57ff6a8 9c278d1496788aee6c7f26556a3f6f2cbb7e109cd20400e0b2381f6c2d4e29f4 (wallet was then empty)
- September 2023: donations come in for Lovera CCS (the only proposal that was in Funding Required)
- September 28, 2023: Luigi logs into CCS wallet to top up hot wallet, finding (after syncing from May 10th as expected) a balance of ~4.6 XMR, representing September donations for Lovera; no additional transfers occurred after September 2
- September 28, 2023 (a few hours later): Luigi has call with binaryFate on what has been discovered; General Fund is confirmed to be intact. Shortly after, Luigi, binaryFate, and fluffypony have a call discussing the situation.
- September 28 - now: Core Team discusses internally; Luigi and fluffypony forensic efforts -- unfortunately, to date, no evidence of breach has been identified
Open questions:
- How do we achieve CCS continuity for existing contributors? Core team is in favor of covering existing liabilities from the General Fund.
- How do we structure the CCS going forward?
- How did the breach occur?
Just to add to this, it's entirely possible that it's related to the ongoing attacks that we've seen since April, as they include a variety of compromised keys (including Bitcoin wallet.dats, seeds generated with all manner of hardware and software, Ethereum pre-sale wallets, etc.) and include XMR that's been swept. See tayvano's thread here. That hack recently started seeing some more sweeps happen (and they can tell that it's from the same hack since the surveillance-chain sweeps go to the same cluster of addresses).
It's entirely possible that other wallets are at risk, which is why luigi1111 and binaryfate have taken additional precautions. I no longer have access to any of these wallets (although I do have large corp / treasury wallets on that laptop that pre-date Monero hardware wallet support and remain untouched), but I've taken similar precautions.
It's also possible that the attacker isn't aware of what they've stolen, in which case I'd ask them to consider that they have stolen funds that are donated by individuals against specific things that Monero contributors are working on. This attack is unconscionable, as they've taken funds that a contributor might be relying on to pay their rent or buy food. I'd urge them to take action to make this right if they become aware of this😞
Shit, thats hard. We've stumbled upon one of the few bad things about crypto that it is irreversible. I can't think of anything other than replacing from the general fund. Also we should use open source hardware wallets like MoneroSigner from now on imo.
"Luigi makes payments from the hot wallet and tops it up from the CCS Wallet (via SSH), occasionally as needed." Does this mean that the private keys for the CSS wallet were on an online Ubuntu server? If yes, thats where the compromise happened imo.
What’s the balance of the general fund, will replenishing the CCS impact protocol development?
Thank you for the transparency and closure about this issue.
shortly after fluffypony's arrest, most of the CCS wallet was swept by Luigi to the hot wallet as a short-term measure pending more information about the nature of the arrest
So to clarify, @fluffypony never had access to the private keys to the hot wallet, but did have the private keys to the main CCS wallet post-arrest?
Would the public be able to get transaction proofs (with addresses) to all nine of those transactions? If the hack was non-targeted, there's a good chance that the receive address gets re-used in someone else's hack, which would help us find the perpetrator.
Going forward, I think that this scenario is an excellent exhibit on why the CCS should use multisig (at least for the main wallet).
So sad to learn about this, please let me know if you need any help for the forensic part.
So to clarify, @fluffypony never had access to the private keys to the hot wallet, but did have the private keys to the main CCS wallet post-arrest?
Yes, as well as keys to the Bitcoin donation wallet, previous Monero GF wallet, etc. Post my release I nuked everything that could potentially be problematic as I was unsure as to what might happen next, and didn't want to put anything at risk.
Would the public be able to get transaction proofs (with addresses) to all nine of those transactions? If the hack was non-targeted, there's a good chance that the receive address gets re-used in someone else's hack, which would help us find the perpetrator.
I'm sure @luigi1111 can do that.
Going forward, I think that this scenario is an excellent exhibit on why the CCS should use multisig (at least for the main wallet).
Yes definitely; multisig was not ready for this prior, but now it is.
@johnalanwoods General fund is around 8k.
will replenishing the CCS impact protocol development?
No, the general fund isn't usually used for funding active development but more for emergencies like this and other unexpected expenses.
There's a clear suspect: https://xmrchain.net/tx/bb77d03cae08942f43cccd759ade505a1c9435470a4a2cabfa5e26d2c93d1a58
Just to add to this, it's entirely possible that it's related to the ongoing attacks that we've seen since April, as they include a variety of compromised keys (including Bitcoin wallet.dats, seeds generated with all manner of hardware and software, Ethereum pre-sale wallets, etc.) and include XMR that's been swept. See tayvano's thread here. That hack recently started seeing some more sweeps happen (and they can tell that it's from the same hack since the surveillance-chain sweeps go to the same cluster of addresses).
It's entirely possible that other wallets are at risk, which is why luigi1111 and binaryfate have taken additional precautions. I no longer have access to any of these wallets (although I do have large corp / treasury wallets on that laptop that pre-date Monero hardware wallet support and remain untouched), but I've taken similar precautions.
The hacks you mentioned @fluffypony were determined to be related to LastPass. This seems to be something different...
The hacks you mentioned @fluffypony were determined to be related to LastPass. This seems to be something different...
A large number of them were, but there are a whole screed of sweeps from users that have never even downloaded LastPass.
"The CCS Wallet was drained of 2,675.73 XMR (the entire balance) on September 1, 2023, just before midnight." Is this midnight UTC or another timezone? If UTC we can assign it low probability to be the same attacker referenced in tayvano's thread:
Primary theft txns are almost always between 10am–4pm UTC
I'm guessing Core is already looking into hiring professional digital forensics specialists, but this could help with prioritizing what data to collect now that might still be around: https://owasp.org/www-pdf-archive//NetSecurity-RespondingToTheDigitalCrimeScene-GatheringVolatileData-TechnoForensics-102908.pdf
Maybe I'm not understanding correctly, but aren't both of @luigi1111 wallets, Ubuntu and Windows, "hot" wallets? Both reside on machines connected to the internet with no hardware devices. Both had their respective spendkeys on them, yeah?
Luigi makes payments from the hot wallet and tops it up from the CCS Wallet (via SSH), occasionally as needed
How was this performed? Did the Windows computer SSH into the Ubuntu computer, or vice versa?
Was the node that the Ubuntu wallet ran on a pruned node or full node?
CCS Wallet Opsec 2.0
- Seed should be generated on the "offline" device
- Only the wallets view key is given to "hot" devices - such that they can generate & broadcast transactions but not sign them
- Wallets are password-protected and/or machines are at-rest encrypted for physical security
- Seed is shared only in encrypted form, and this wallet setup must replicated by whomever holds a copy of the spend key
- Key images and outputs are transferred in the same way as transactions
The offline computer could be a scrappy $200 notebook, what's important is that it is offline forever.
There is a burden when moving funds like this, but then again - this is a large amount of community funds.
Having more "hot" buffers would spread out risk as well, and would speed up the payout latency for contributors, e.g, @plowsof could be given enough funds to pay out soon-to-be-finished CCS's (assuming he doesn't vanish)
Core team is in favor of covering existing liabilities from the General Fund
Now that this is disclosed, current contributors who have been waiting for payment should be paid ASAP :)
Now that this is disclosed, current contributors who have been waiting for payment should be paid ASAP :)
Core and their helpers have often been trying to pay things out over the years. But a combination of some people being unreachable, refusing payment, or other such circumstances means that funds often sit there. Many times for years. It may be wise to institute a form of expiration policy where unclaimed funds (x months or years after funding/project completion) go into a special "Fund other CCS projects" wallet or something. All of this Monero sitting there years after funding are a liability.
Maybe I'm not understanding correctly, but aren't both of @luigi1111 wallets, Ubuntu and Windows, "hot" wallets? Both reside on machines connected to the internet with no hardware devices. Both had their respective spendkeys on them, yeah?
Luigi makes payments from the hot wallet and tops it up from the CCS Wallet (via SSH), occasionally as needed
How was this performed? Did the Windows computer SSH into the Ubuntu computer, or vice versa?
Was the node that the Ubuntu wallet ran on a pruned node or full node?
Windows -> Ubuntu, once every 3 months or so. Full node.
Windows -> Ubuntu, once every 3 months or so. Full node.
"Luigi makes payments from the hot wallet and tops it up from the CCS Wallet (via SSH), occasionally as needed." Does this mean that the private keys for the CSS wallet were on an online Ubuntu server? If yes, thats where the compromise happened imo.
I am of the same opinion. All Tayvano's "OG" friends were also Windows users and considering the amount of well done and undetectable malware existing for that OS, I wouldn't be surprised if Luigi's Windows machine was already part of some undetected botnet and its operators performed this attack via SSH session details on that machine (by either stealing the SSH key or live using trojan's remote desktop control capability while the victim was unaware). Compromised developers Windows machines resulting into big corporate breaches is not something uncommon.
A first step to investigate this is to log that machine's network traffic on the router that connects it to the Internet. A log time should be at least 48 hours (but more = better) with any software using network switched off to maximize the log's quality by reducing the noise to the possible minimum. Backdoors existing today are capable of being very low profile in terms of networking and detecting them isn't easy, therefore it will require some time and patience.
This is the only possible realistic attack vector in this case, given that the timeline provided in the OP doesn't omit some more important information.
P.S. beware that chances to discover the malware are 50/50, given that the attacker may track all the public communications related to this event including reading this thread, who could decide to detach/deactivate the backdoor to clear the evidence and avoid its disclosure. So consider making a full disk dump of that machine as well.
P.P.S. stop using Windows for such projects.
The attacker likely consolidated the funds again in these two transactions. Exchanges and services should check to see if they received these XMR deposits.
https://xmrchain.net/tx/2c5b45bf398dcae482019a46fb2d06d334bf4260484dc4857fc35db3689ad5ec
https://xmrchain.net/tx/06550272cdfa1eea98d288b2d57c272b5c52a2b195b4f808c8c03422a58ca47b
I think that nobody asked that before, @luigi1111 I have few questions about the Ubuntu server
- Was it running at your place (i.e. phisical device you had access to that was being turned on when needed (especially: was offline during the 'Incident'))
- If not, was it a dedicated cloud server or a KVM/OpenVZ/other VPS (if possible, tell us who was the cloud provider)?
- Which version of Ubuntu was it running at the time of Incident?
- Was the Ubuntu server accessed via SSH password authentication or key?
- Lastly, not a question - but if you have logs of any kinds (maybe logs in backups), try securing them, if it was a cloud server download oldest possible backup(s), and grab copy of all server logs.
p.s. @SamsungGalaxyPlayer are you tracking monero 😕?
I think that nobody asked that before, @luigi1111 I have few questions about the Ubuntu server
- Was it running at your place (i.e. phisical device you had access to that was being turned on when needed (especially: was offline during the 'Incident'))
- If not, was it a dedicated cloud server or a KVM/OpenVZ/other VPS (if possible, tell us who was the cloud provider)?
- Which version of Ubuntu was it running at the time of Incident?
- Was the Ubuntu server accessed via SSH password authentication or key?
- Lastly, not a question - but if you have logs of any kinds (maybe logs in backups), try securing them, if it was a cloud server download oldest possible backup(s), and grab copy of all server logs.
p.s. @SamsungGalaxyPlayer are you tracking monero 😕?
- It was running at my place.
- n/a
- 20.04
- Password
P.P.S. stop using Windows for such projects.
If you are truly concerned about malware, simply switching to Linux isn't a great answer. Default Linux installations are not that great for security and not very hardened. You need a hardened system, preferably an immutable OS that has the root partition as read-only, IE Fedora Silverblue or any other OSTree based systems. Use https://cisofy.com/lynis/ to see any potential unnecessary security issues and things that weren't being used that can be turned off. Setup automatic updates. Only use Wayland, as X11 is easy to keylog. Use keys for SSH, not passwords. Or better yet SSH turned off. If you need to access it do it physically.
Same thing goes for the CCS node/wallet server. Using UEFI Secure Boot, LUKS encrypted main, root, and GRUB partition. Wanna get crazy you can do coreboot with heads on some specific systems that support it. Don't use LTS kernels, use the latest one with grsecurity patches. Just suggestions.
Also a given, these two devices should be VLAN'd from the rest of the network if not already.
@hinto-janai 's model would already greatly improve what already exists, offline signing would take so many potential attack vectors away.
Also secure the network if not already. Run an OPNSense firewall to VLAN and make sure no unnecessary ports are open. Use an OpenWRT router if you need wireless. Countless shitty consumer routers don't get updated ever, and many of them have severe vulnerabilities that don't get patched for a really long time.
P.P.S. stop using Windows for such projects.
If you are truly concerned about malware, simply switching to Linux isn't a great answer. Default Linux installations are not that great for security and not very hardened.
I didn't say one should use a default Linux installation. What you said should be already obvious to people with such responsibilities. What's surprising is that this is being explained to people from Monero team.
I didn't say one should use a default Linux installation. What you said should be already obvious to people with such responsibility.
Fluffy's setup was much better..
@luigi1111
It was running at my place. Was it exposed to the public internet in any way, other than your laptop or only available via LAN?
I think that this may be the most likely cause of the incident, I doubt someone 'guessed' the seed right.
Fluffy's setup was much better..
Yeah it corresponds to the industry standard where the threat agent is LE.
I didn't say one should use a default Linux installation. What you said should be already obvious to people with such responsibilities.
Given that Windows was being used these things probably aren't obvious. Most people are not very knowledgeable on the inherent security issues with desktop operating systems, or basic hardening.
I'm not 100% caught up on this thread yet (just getting back home) but here's some more specific details on the threat actors ive been chasing for a good while now:
typically operate 1200 utc - 2300 utc, though all hours have been observed. least amt of activity 300-1000 utc
observations we have on them for the time period mentioneed by op:
2023-Aug-30 21:50 2023-Aug-31 13:09 2023-Aug-31 18:29 2023-Aug-31 18:31 2023-Aug-31 20:13 2023-Sep-03 12:31 2023-Sep-03 12:32 2023-Sep-03 12:35
for those above timestamps, all activity was via
2a00:1650:0:3:45::1
2001:ac8:23:3c:2d4::1
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
These are either HideMe VPN or residential proxies (5socks etc), as is usual for these actors.
Victims have all sorts of devices. The only real thruline is the age of keys (created prior to 2022, some keys date back as far as 2012) and everyone we have talked to has used lastpass, most have confirmed the specific keys/seeds that were compromised were in lastpass, usually a secure note, at some point. most are longtime lastpass users but a few only used last past for a short period of time. those users confirmed the specific compromised keys were in lastpass.
fwiw, these actors even push stolen XMR to BTC—we have observed them consolidating a victim's eth, btc, xmr to a btc address before pushing to wasabi/sinbad/cryptomixer/coinomize/etc. they use instaswappers to do so e.g. fixedfloat, simpleswap, sideshift, etc. The size of this theft should make the funds easily findable in the outgoing transactions from the hot wallets of those instaswappers on BTC if it is in fact these same threat actors.
The first thing I would ask is if anyone who's ever had access to the keys that were compromised here has had other wallets drained in the last ~year. Even if the amt stolen from those wallets was small / dust. That will help determine source of compromise faster than anything else tbh.
I can't understand why on Earth you would use less secure system (the Windows hot wallet) to SSH into the what is supposed to be the more secure system (Ubuntu). With a password no less.
Neither do I understand why you would choose Windows or Ubuntu for either operation in the first place. If you're not an expert at sysadmin and security, then you should be using Qubes for this amount of funds, and/or offline key storage.
I am of the same opinion. All Tayvano's "OG" friends were also Windows users
@marcovelon You know, I'd really like to see some citations in a thread about cybersecurity when even Tayvano said all sorts of devices.
My questions for Luigi and Fluffy Pony are:
- Where all has the CCS seed been stored (who, device, app, file format)? e.g. Windows/ubuntu, Password managers, text files, etc.
- Also, If the Ubuntu server storage got shot one day, how would you restore the CCS wallet?
- Does fluffypony have the means to restore the wallet himself (as in could a hack on him lead to wallet drain)?
I am of the same opinion. All Tayvano's "OG" friends were also Windows users
@marcovelon You know, I'd really like to see some citations in a thread about cybersecurity when even Tayvano said all sorts of devices.
My questions for Luigi and Fluffy Pony are:
Where all has the CCS seed been stored (device, app, file format)? e.g. Windows/ubuntu, Password managers, text files, etc.
- Also, If the Ubuntu server storage got shot one day, how would you restore the CCS wallet?
Seed was only on paper on my end.