CCDC: Tips and Tricks to Managing a Team
How to Succeed at CCDC
What is CCDC?
Last April a friend of mine whom I had been working with in a class at Purdue recommended I join him for a club meeting. When I asked what the club was for he said CCDC or "Collegiate Cyber Defense Club".
If you're not familiar, Cyber Defense Competitions are competitions where a Blue Team defends a network from a professional Red Team. These small-scale networks include multiple operating systems, services, and configurations that would be seen in a real world environment. Most of the time, these systems are older and extremely vulnerable.
As the competition proceeds, points are awarded for keeping services online as well as responding to business scenarios, much like a corporate IT team would face. Scenarios might include allowing VPNs from home, implementing IT policies, and building incident response reports.
Our team competed in state and wild card levels of the NCCDC -National Collegiate Cyber Defense Competition hosted by Raytheon. You can learn more about their competition on the website.
Managing a Team & Preparing for competition
I was privileged to serve as President of the club after a short time period in the club. Over the course of a year of planning, administrating, and marketing the club; here is what I learned and what I took away from the club. If you're looking for my team's scripts and plans, you'll be disappointed; sorry to those doing OSINT on other teams ;)
Please note, this was only my first year competing in blue team style competitions. This article serves to note what can be learned, focused on, or if you do not have any experience in such competitions; what it is all about! I took a management and director role within the team and I can best speak on these type of responsibilities.
1. Define strengths and weaknesses of your team early
Our team had been competing for the first time. We had a general idea of how the competition would go, but we did not have years of experience as some of the other teams did. What went well was that early on in April, before planning the agenda for the year, we mapped out what we knew and what we did not know.
CCDC in particular includes hosts that are running Palo Alto, Windows, Linux, as well as all of their respective services. During our competition we saw DNS running on both Windows & Linux (Bind), mail servers (Post Fix), Active Directory & GPO, Websites and their Applications (including their back-end SQL databases, as well as an IOT device.
With these wide variety of concepts, it's good to map out what your team is good at or is not good at so that they can seek resources to learn more throughout the year. With your weaknesses cataloged, a general curriculum can be planned.
2. Recruit members for their skills and strengths
The harsh reality of the nature of the competition is that a school can have 8 students compete. For some schools this is not a problem. For others, like a large research university, it becomes very difficult to include friends or all those interested. There are a lot of large clubs which are great for networking and sharing interests, however it quickly becomes a social environment and hardly a place to practice or learn new skills.
To combat this, our team worked on a referral / word of mouth recruitment style. Our team was split between Cybersecurity / Networking and Computer Science students. When group projects went well or when a student stands out in a class, one of our team members would approach them and invite them to the team. Students approached would then interview with the club and be asked about their interests, skills, and weaknesses. After, the club would decide to admit or deny the member.
Keep in mind you want to recruit to fill the gaps in where your team may be missing knowledge or experience. The rules for CCDC allow a graduate student on the team. This can be a huge advantage to fill a large hole in your team.
3. Keep the team small
Many clubs on college campuses seek to get as many numbers as possible, to network to promote their cause, and spread or teach others the knowledge of what ever topic the club revolves around. This is great, if you were not limited to 8 people.
Keeping your team small allows for the team to get to know each other. We purposely would not have call outs and rely on our word of mouth recruitment. We aimed for 12-15 members so that we had a good pool of talent for the competition, but were not over crowded.
We also accepted the harsh reality that not everyone in the club, will get to compete in the final competition. Our members were asked if they were okay with this. It added an element of competition to the club and ensured only the best and most active members were competing. This can be the most difficult aspect of the club. Ours was ran in a bit of an aggressive manner; the team wants to win.
4. Develop specialized sub-teams
There is a myriad of skills that will be required to be successful in a cyber defense competition. In reality, one single person cannot learn all of those skills required. Our team elected to build sub-committees broken up into three areas: Windows, Linux, Firewall.
These sub-teams allowed those with interests and skills aligned to apply themselves to a specific area. It also specialized our team by allowing team members to focus on skills specific to their team. No longer did someone have to spend time learning powershell or group policy while still trying to learn the firewall used in the competition.
Our computer science team members who were very familiar with unix were able to dive directly into it and start learning the more obscure distributions. They were also ready to begin building scripts (more on this later).
However, this can be dangerous...
5. Don't rely too much on specialized sub-teams
If your team develops these specialized teams, ensure that the teams do not become too isolated in their environments. Should something happen and a team member not make the competition and require an alternate, sub-teams may shift. If your linux team can only do linux and one of its members now find themselves doing group policy on the Windows team, your team is now at a serious disadvantage.
We incorporated stand ups. Every week or two, each team was responsible for taking a few minutes out of the meeting and telling us what they are working on, what their challenges are, and demo any scripts or programs they're developing. This ensured that all teams are aware of what the others are up to and when a plan goes wrong, other team members are aware of the sub-teams processes.
This assisted us first hand when we had to compete in a wild-card competition during spring break. One of our team members had already left for vacation and we needed to sub in a member. Thankfully, we ensured all members were up to date on each others work so the substitution could be made with little headache.
6. Get Industry and Faculty involved
CCDC style competitions are growing in popularity. It is a great way for students to experience what incident response and system hardening is like in the real world. Many tech companies and firms are getting involved as a way to scout talent.
Our team reached out to several companies and firms that members had previously had internships with. Many of these companies sent consultants to lecture or demo tactics that their Red or Blue teams used in the real world. Some of these consultants even had prior CCDC experience and could offer directly their experiences and lessons from their time competing.
Inviting consultants is an amazing way to get more information for the competition, but to also network. Students without internships may benefit from getting in direct contact with companies interested in hiring security minded students. Companies will also benefit from finding those with experience!
Faculty is also a great resource. In our case, we asked a professor who taught a class on firewalls to come lecture at our team meetings on firewalls. We also had professors with backgrounds in Network Administration, Incident Response, and Forensics all give lectures and advice on the competition.
7. Build, troubleshoot, and destroy a test network
One of the best ways to learn networking and virtualization is to simply do it. Our team built the topology from previous years competitions after purchasing some cheap hardware. Doing so allows you to understand all of the problems that will come along the way.
Understanding how DNS, DMZ, Firewall, ESXi work and function will simply add to your knowledge of security. When your team goes into the competition, they will have a stronger knowledge of the variations of the topology because they have actually built one! There are great resources online that walk you through the process of implementing these critical systems.
Once you have built the network, you can also learn how the Red Team may think. When you have faculty or industry consultants come to your meetings, give them a chance at looking at your network. Allow them to attack it and explore its weaknesses. That's the beauty of virtual hosts, you can roll it back and do it over and over!
8. Learn how a service works before you secure it
This may sound incredibly straight forward, but it is one of the key improvements our team identified for future seasons. Trying to troubleshoot or secure active directory, post fix, or bind is difficult on its own; even more so if you do not understand how to configure or implement the service.
Take time to learn how the services work. Again, the test network is a great place for this to be done. Implement difficult or complicated systems such as mail servers, e-commerce websites, DNS (Bind), and then see what breaks them or troubleshoot and take notes.
9. Script.... but don't
If you have never competed in a CCDC style event, there is a lot going on and it is the fastest 5-6 hours of your life. To help manage time, the development of scripts is a good idea, but use at your own risk. Scripting allows for things to be done at the press of a button and while it runs, your valuable time can be spent elsewhere.
However, don't assume the scripts will work universally. CCDC is a realm where the unthinkable becomes reality. We have seen hosts where apt-get or yum was uninstalled, package lists were broken, and a plethora of other problems. Since you have no idea what the host's condition will be in when you arrive, don't rely on your scripts.
Some things are better left to be done by hand. IP tables and BIND are very touchy services. Setting a script lose on the network can be dangerous if untested.
Enduring the Competition
While the list of these topics are not in really any order, this one is a huge one. Too much or too little communication can be a critical mistake during the competition. As services go offline, scripts fail, or new unfamiliar services are encountered you will need to rely on your team. With 8 of you in a room, there is a chance someone has experience in what you're dealing with. Ask for help.
During our competitions we stationed ourselves by sub-teams to allow for better communication between members. We also sat so we were close to each other and can quickly move around to see each other's screen or assist with problems.
This being said, keep communications clear when unnecessary. Jokes and side conversations can wait. It is an extremely intense competition and you do not want to raise tensions between team members because someone is not taking the competition seriously.
2. Have a plan, but don't rely on it
One of the best things about the practice times are teams can practice their strategies, see the network, and run their tools on the network. This being said, don't rely too much on the practice network being exactly the same.
Your plan or strategy should contain high level ideas and a checklist of what should be one. For example, "Change passwords" not "Change root password on Mint Host to: thisisagoodpassword1!". This ensures that you will not be following the plan religiously as when something breaks, you will be more adaptive to change. Similar to not relying on scripts too much, you want to have a plan in place but also be ready to abandon it.
3. Logs and Notes - Debrief
One of our biggest strategies going into our first competition was to find ways to improve for future competitions. Taking notes during the competition both on a personal and team level are a great way to reflect later when the dust settles. Debriefing after the competition and taking notes on what went well, or more importantly what didn't, will allow you to adapt and train for either the next round of the competition or for the next season.
Depending on the rules of the competition, exfiltrating logs or configurations is a good idea to actually take time and learn where the vulnerabilities lay. For example, pull Bind config files and save it to a google drive so when the competition is over, and you can breathe, it's possible to go through the configuration file and see what you may not have during the intense competition.
Overall the whole point of the competition is to practice and learn new skills relating to defense. This is impossible to do if you cannot build a scope of what needs to be accomplished.
Have fun and immerse yourself. I will be graduating in the fall and I am sad I could only compete with this team for one season. However, I learned more in the year of preparation than I have in any one class. It bridges together your course studies as well as industry knowledge and as you go through the rounds of challenges like CCDC, you're matched against the creativity of both the Red Team and other schools. It's as close to the real thing as you can get.
Network! CCDC is a growing competition and has a lot of resources available. Mentioning the competition at conferences to vendors and other attendees often results in tips and tricks and if you're lucky internships and employment opportunities. There are a lot of people who have previously competed in the competition and would love to offer assistance.
If you haven't already, I highly recommend trying one of these competitions. To my team that is reading this, thank you for an amazing year!