Sometimes people like to tell me that multi-user systems are dead and that no one uses them, which always is funny to me, but it’s less funny when a really nasty bug pops up that breaks things because of it.
At one of my previous shops we managed a few servers for password cracking that were essentially servers with a small pile of graphics cards, some decent disk space, and a very old school problem. We all had to share and queue up our password cracking jobs into hashcat for the 30+ pentesters dumping hashes on a weekly basis.
This is a pretty traditional workflow, ssh in and trigger your jobs with some wrapper scripts to queue up your jobs and prevent job stealing. Being good hackers and trying to isolate jobs execution from each other (and other client data) we used user and group seperation and the job controls were handled by a SETGID directory with a group that all testers belonged to.
Things always worked smoothly for the most part, but every once in a while you would have your pile of NTDS.dit hashes to crack, the right hashcat arguments, and a dream. But then you’d run into an issue, a simple Permission denied on a OCL kernel file and hashcat unable to generate it’s kernels to optomize cracking causing it to bail.
Apparently, this had been going on for a long time since I had been around and I had always run my own cracking servers so never had an issue with this. The current “solution” was to wait for someone to complain and then run an script to automatically clean out the kernel files and in some cases some aggressive chmod’ing. When I overheard that the next-generation solution was to use cron or some other triggerable fixes with sudo I couldn’t handle it anymore.
After someone complained about the issue again for the nth time this week I got out my old sys admin hat and started digging. After recording the exact command they were running, switching user contexts, and hucking some debugging options at hashcat the problem became pretty clear. When exact parameters for optimization were passed to hashcat and the hash type was the same hashcat would create the kernel in the correct directory with their username and the group set to the correct shared group. But it was obvious, the permissions on the OCL kernel were 700 which meant that if someone passed the exact same arguments and a kernel was attempted to generate, it was being done so as the shared group and would fail since there is no group write permission allowed on the file.
Simple.
Now just to figure out where hashcat is doing this at. So I grabbed the source code and did the logical thing: grep -i -R uname\( *
The only result was in the setup_umask function located here. Which was the following bit of code:
void setup_umask ()
{
umask (077);
}
Well… that would do it. This function was only ever called from the main function during set up of the jobs after building contexts and had no arguments to disable the thing.
Generally, it is considered poor form to not use the user umask(2) and change it on a per-file basis. I was wondering where this came from when I decided to search the GitHub issues and found this issue from 2016 that was resolved with the quite funny line: I've set the umask now. Let's see if someone complains :)
Of course, being the good internet citizen I commented on the massively dead thread and moved on.
Now that I knew the issue and how to fix it patching would be simple right? I wish. The problem is that the system is managed in the traditional Penentration Tester™ method of 10 system maintainers and having hashcat installed from package management. There was some push back to me suggesting having to set up a pipeline and patch it inline for new builds since that was “quite a lot to manage for a one line change” and I was told I should try and fix it from the management script side, so I was left with few options.
Then I had a terrible idea.
There was only one call to umask(2) and at the time had just wrapped up a Go userland LD_PRELOAD rootkit proof-of-concept. So the solution should be obvious now. Of course I wrote into the system management scripts a few short calls to GCC, a copy, and changed all the calls to hashcat to prepend LD_PRELOAD=/usr/share/hashcat/umask_hook.so:
#define _GNU_SOURCE
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <dlfcn.h>
mode_t (*__umask)(mode_t)=NULL;
mode_t umask(mode_t cmask) {
printf("!!! umask(2) hook applied !!!\n");
if(!__umask)
__umask = dlsym(RTLD_NEXT, "umask");
mode_t og = __umask(0017);
return og;
}
Here is the project listing that I threw into a repo if anyone is interested.
I actually ended up needing this technique a few other times to fix other programs with similar issues and another for hashcat again. I ended up taking quite a few lessons away from this quick little distraction.
Techniques learned from attacking can be very useful in strange places. At this point I think I know how to manage Windows better from Meterpreter than from a real workstation.
And most importantly don’t do this - it’s crazy.