What a tough day for your organization…
Your shipping clerk receives an email from a “customer” asking for review of a pdf invoice. She feels this is an unusual request, but it looks official. She decides to look at the invoice because… that makes sense…you want to know what you are taking to the supervisor to answer questions she may have. The supervisor looks at the email and lets the clerk know its just a piece of spam. Lucky, we caught this before we paid money to these scammers!
Great work by the shipping team! I mean, they are trained to follow processes and they know which customers are legitimate.
Midnight that following weekend your IT team receives several tickets. “We can’t login to anything. Could you check the domain controllers?” Your on-call IT knows this problem too well. “I bet this is a storage issue again. I told management to upgrade these arrays last year. I can’t wait until Monday to unleash my I-TOLD-YOU-SO!”
She VPNs into the network and logs into the primary domain controller. Her laptop becomes slow to respond. Her applications are not starting up or are crashing. What in the world is happening here? She opens her My Documents folder to grab the IT director’s phone number and sees an odd extension appended to every file. These files are not accessible. Her heart begins to race as she now knows exactly what is happening. The files are encrypted, IT’S RANSOME WARE.
She takes steps to remove all servers and workstations from the network, she contacts the network administrator to remove any possibility of spreading to your other sites. It’s all hands on deck. This is going to spread. The team begins showing up on site to triage. The extent of damage is the worst. It is too late. Every server vetted so far is infected, every workstation is infected, files in shared storage are all encrypted. Panic sets in as the IT Team begins thinking about resumes and job opportunities.
This very real scenario has happened to many companies across the world. Production facilities ground to a halt and unable to fulfill orders. The last days of a business. The last attempts to salvage the company by cutting checks to criminal organizations that have no incentive to deliver the solution.
The minds at God Particle IT Group were able to save one such organization. Years of expertise from handling catastrophic failures such as this saved this company from losing tens of millions of dollars. The affected company saved themselves by turning to professionals.
Our engineers were boots on the ground before hour two. By working with business leaders, we determined that saving application and customer data were top priority. A lot of these systems relied on legacy, in-house applications of which the original developers were no longer employed and out of reach. The databases in these applications held critical, irreplaceable configuration information. Not to mention other databases which held customer orders and product information.
This facility stood to lose millions of dollars a day for the duration of halted production. There was the requirement to recover application and order data, but equally as important was to get the facility back into production to keep the company alive.
God Particle engineers quickly strategized and came up with a plan to move efficiently. We dispersed into teams to first isolate bad systems. There would be legal ramifications for this company if an investigation were performed poorly. The evidence had to remain intact. To ensure preservation of the “crime scene” we isolated half of the network physically and began wiping configuration for the network infrastructure to build a “clean network.” Once the clean network was in place, we would have an area to begin implementing new, clean infrastructure.
A major hurdle to restore services was the need for clean equipment. We could not use the existing equipment due the need to preserve the data. Waiting for equipment orders would cost millions. We were able to scavenge 3 UCS hosts and a Nimble storage array. This would allow us to begin building a new environment for applications immediately. Not a permanent solution, but one that would get the facility producing again.
The next major hurdle was the restoration of critical data. The company had backups…local backups. To local storage. The same local storage that was encrypted by ransomware. The company’s executive team was in a state of shear panic. Our engineers determined that since the databases were in an active/passive type of failover configuration, there was a chance that the standby DB may not have been affected. We were also aware that logging on to this database may potentially trigger the ransomware to being encrypting data. Our solution was to attempt a recovery and restoration of one of the mirrored drives in the DB server.
The executive team was overjoyed at the news …we successfully recovered the data that would keep the company running. I cannot tell you how proud we were of this accomplishment as well. Mishandling of those database servers, someone carelessly trying to login or evaluate without consideration, could have destroyed the initiative.
We spent the next few days standing up a new domain since the old was irrecoverable. Over 500 workstations were reimaged and hardened. We provided a design that would isolate areas of the network and help prevent a quick and ultimate spread of malware like this again. We worked with the company’s in-house development team to restore the applications to a new redundant host environment.
The company’s endpoint protection, which failed to execute and isolate this attack, was upgraded, and implemented properly. A new security architecture was agreed upon by the business leaders and operations managers.
Within a week the facility was back in production. No money was sent to criminal organizations. A much larger disaster avoided. There was definitely still work to be done but no longer at an emergency pace. Managers and executives could sleep again.
Prior to the incident, the company made efforts to establish a secure network. Most companies these days understand the value of security, but may not have the manpower, budget, or skill to replace or harden systems. Threats constantly evolve. If skills aren’t growing to match the evolution, the threats will eventually overpower unmanaged security implementations. This was such a case. The IT team for this organization was very competent. All of the boxes were checked… backups, firewalls, endpoint protection, a modest bit of user training. Yet the threat still damaged the organization greatly.
Overall, this company paid a hefty price and lucked out of tanking. God Particle played a great role in this. Our engineers worked tirelessly. An overused term for sure but very appropriate in this situation. An exhausting but very rewarding project where we came out victorious.
Please reach out to us. We would take great pride in evaluating your infrastructure and creating solutions to ensure 1 event will not ruin your organization. And if you happen to be at the point where it is too late, we firmly believe our engineers are your best shot.