Pipal, Password Analyser

On most internal pen-tests I do I generally manage to get a password dump from the DC. To do some basic analysis on this I wrote Counter and since I originally released it I've made quite a few mods to it to generate extra stats that are useful when doing reports to management.

Recently a good friend, n00bz, asked on Twitter if anyone had a tool that he could use to analyse some passwords he had. I pointed him to Counter and said if he had any suggestions for additions to let me know. He did just that and over the last month between us we have come up with a load of new features which we both think will help anyone with a large dump of cracked passwords to analyse. We also got some input from well known password analysts Matt Weir and Martin Bos who I'd like to give a big thanks to.

I have to point out before going on, all this tool does is to give you the stats and the information to help you analyse the passwords. The real work is done by you in interpreting the results, I give you the numbers, you tell the story.

Seeing as there have been so many changes to the underlying code I also decided to change the name (see why) and do a full new release.

So, what does this new version do? The easiest way to explain this is to show the output generated by parsing a leaked password list. I've chosen the list of passwords from the phpBB leak which I grabbed from the SkullSecurity site.

The first output is the number of entries in the file parsed and the number of unique entries found. Unfortunately the list I chose has already been ran through unique so these two figures match in this example.

Total entries = 184373
Total unique entries = 184373

The top 10 passwords. In this situation the list I chose has already been passed through a filter to strip any duplicates, this is why each word only appears once. The cap of showing the top 10 is configurable by a parameter on the command line, I'd suggest playing with this limit as sometimes the next entry is the one that starts to explain things.

Top 10 passwords
123456 = 1 (0.0%)
password = 1 (0.0%)
phpbb = 1 (0.0%)
qwerty = 1 (0.0%)
12345 = 1 (0.0%)
12345678 = 1 (0.0%)
letmein = 1 (0.0%)
111111 = 1 (0.0%)
1234 = 1 (0.0%)
123456789 = 1 (0.0%)

The next list is the number of base words. I define a base word as a word with any non-alpha character stripped from the start and end. This is useful to identify common words such as company names or places which the passwords have been based on. I did consider stripping all non-alpha but in one of the lists I tested on I found the base word "un1c0rn". Leaving the non-alpha in the word makes sense, removing them you get "uncrn" which doesn't really mean anything.

Unsurprisingly as this list came from phpBB the top word that passwords are based on is "phpbb", "password" next is another obvious base word but then "dragon" is one I wouldn't have expected.

Top 10 base words
phpbb = 332 (0.18%)
password = 89 (0.05%)
dragon = 76 (0.04%)
pass = 70 (0.04%)
mike = 69 (0.04%)
blue = 67 (0.04%)
test = 66 (0.04%)
qwerty = 59 (0.03%)
alex = 58 (0.03%)
alpha = 53 (0.03%)

Lengths are next, fairly self explanatory. It is a shame that the people who put the effort in and had greater than 20 character passwords still got theirs leaked.

I hope that the 948 three and under words are a mistake made when cracking the list.

Next a nice graph showing the length data, I'm quite proud of getting this in.

        |                                                               
        |                                                               
        |                                                               
      | |                                                               
      | |                                                               
      | |                                                               
      |||                                                               
      |||                                                               
      |||                                                               
      |||                                                               
      ||||                                                              
      ||||                                                              
      |||||                                                             
     ||||||                                                             
    ||||||||                                                            
|||||||||||||||||||||||||||||||||                                       
000000000011111111112222222222333
012345678901234567890123456789012

Some more self explanatory information comes next. 30% of people chose a 1-6 character password and 40% chose one that contained only lowercase alpha characters.

One to six characters = 55807 (30.27%)
One to eight characters = 143874 (78.03%)
More than eight characters = 40507 (21.97%)

Only lowercase alpha = 76041 (41.24%)
Only uppercase alpha = 1706 (0.93%)
Only alpha = 77747 (42.17%)
Only numeric = 20728 (11.24%)

First capital last symbol = 225 (0.12%)
First capital last number = 4749 (2.58%)

The external list is a list of words passed in to Pipal on the command line. I check how many times each of these words is included in each password. This is similar to base words but here you tell the app which base words to search for.

If you are wondering why "dragon" is only counted 76 times as a base word but shows 185 times here, that is because there are 109 base words which contain "dragon" but aren't just "dragon", for example "phpdragon".

The external list I'm using is the list claiming to be "The 25 Worst Passwords on the Internet". Another suggestion for a list of words to use is the domains from the Alexa top 1000 list, this could be good if you are analysing a list of passwords from an unknown origin or would like to know if a list from one domain is linked to any other domains.

External list (Top 10)
master = 229 (0.12%)
123456 = 208 (0.11%)
dragon = 185 (0.1%)
password = 164 (0.09%)
monkey = 118 (0.06%)
shadow = 105 (0.06%)
qwerty = 95 (0.05%)
1234567 = 72 (0.04%)
12345678 = 47 (0.03%)
letmein = 44 (0.02%)

We now look at months and days in both full and abreviated form. While "may" could be a persons name or normal word it looks like for some reason it is a popular word in the list. "June" and "April" are also popular but also names which could explain the higher proportion. For days of the week there is a very large preference for "monday" and "friday", guess which days people change their passwords.

Seeing as we've looked at months and days why not years. Looks like years around the turn of the milenium are popular in this list. I also ran this on the passwords from the myspace leak which showed years around 1990 were popular, maybe this says something about the age of the average user.

The common assumption is that when people are foced to use passwords with numbers in their general response is to add a single digit on the end. Looking at this next set of stats, in this list people actually prefered to add two digits onto the end. The assumption that the last digit will be "1" does however hold true.

 |                                                                      
 |                                                                      
 |                                                                      
 |
 |
 |||
||||
||||
|||||||| |
||||||||||
||||||||||                                                              
||||||||||                                                              
||||||||||                                                              
||||||||||                                                              
||||||||||                                                              
||||||||||                                                              
0123456789

We now look at what the last digits are. Some of the numbers are expected but others, 21984 for example, aren't. Could this be a US zip code?

These last three are recommendations from Martin. These are where we start moving from analysis to cracking, character sets and hashcat masks.

Install / Usage

The app will only work with Ruby 1.9.x, if you try to run it in any previous versions you will get a warning and the app will close.

Pipal is completely self contained and requires no gems installing so should work on any vanilla Ruby install.

Usage is fairly simple, -? will give you full instructions:

$ ./pipal.rb -?
pipal 1.0 Robin Wood (robin@digininja.org) (www.digininja.org)

Usage: pipal [OPTION] ... FILENAME
        --help, -h: show help
        --top, -t X: show the top X results (default 10)
        --output, -o : output to file
        --external, -e : external file to compare words against

        FILENAME: The file to count 

When you run the app you'll get a nice progress bar which gives you a rough idea of how long the app will take to run. If you want to stop it at any point hitting ctrl-c will stop the parsing and will dump out the stats generated so far.

The progress bar is based on a line count from the file which it gets this using the wc command. If it can't find wc it will make a guess at the number of lines based on the file size and an average line length of 8 bytes so the progress bar may not be fully accurate but should still give you an idea.

Download

Download Pipal 1.1 - Bug fixed, not calculating correct percentage for Hashcat masks - Reported by Moshe Zioni
Download Pipal 1.0

Analysis

This section was supposed to just contain a few sets of sample stats but as more sites are being hacked and passwords released I've decided to run analysis on any lists I can get my hands on and post the results here. The first six in the list are the original sample sets and are based on password lists from the SkullSecurity site, for the rest, I'll give whatever information I can about where the list came from.

• phpBB
• myspace
• Hotmail
• RockYou top 1000
• The full RockYou list - Took over 24 hours to generate
• Faithwriters
• Yamaha MotoGP - This one is from here
• CSDN - For more information and a download link see The Hacker News Network
• T-Mobile - A small leak, barely worth analysing but hey, bad passwords are bad passwords and these are bad passwords! For more information and a download link see T-Mobile staff data and passwords hacked and published
• Irish Aid - Another small leak, this time from the Irish Aid website. See the story on Help Net Security
• Portal Mercosur - This is from a database dump posted on Pastebin
• Foxconn had a big leak, here are three different lists from them Partners, Users, Vendors. The data can be found as a torrent on The Pirate Bay.

The following stats have been generated by other people.

• TeamPoison UN email passwords (http://pastebin.com/FEcE9WzJ) with blank pwds (000) - Generated by @Rob_OEM
• TeamPoison UN email passwords (http://pastebin.com/FEcE9WzJ) 270 blank pwds removed - Generated by @Rob_OEM
• Specialforces.com - Damian cracked these passwords and performed the Pipal analysis. Read his write up here.
• "GoD zERo" zine Disney passwords - As cracked and analysed by N00bz.
• Stratfor - Cracked by Martin Bos of Question Defence.
• Stratfor version 2 - This one was cracked and analysed by Marc Doudiet of Swiss infosec.
• Stratfor version 3 - Everyone is cracking Strafor passwords, this one is a full write up by Electric Alchemy.
• Loads of analysis - @arex1337 has done an awesome job of collecting together and analysing a whole host of password lists.

Feedback/Todo

If you have a read through the source for Pipal you'll notice that it isn't very efficient at the moment. The way I built it was to try to keep each chunk of stats together as a distinct group so that if I wanted to add a new, similar, group then it was easy to just copy and paste the group. Now I've got a working app and I know roughly what I need in the different group types I've got an idea on how to rewrite the main parser to make it much more efficient and hopefully multi-threaded which should speed up the processing by a lot for large lists.

I could have made these changes before releasing version 1.0 but I figured before I do I want to get as much feedback as possible from users about the features already implemented and about any new features they would like to see so that I can bundle all these together into version 2. So, please get in touch if there is a set of stats that you'd like to see included.

One other thing I know needs fixing, Pipal doesn't handle certain character encodings very well. If anyone knows how to correctly deal with different encoding types, especially with regards to regular expressions, please let me know.

Where is the name from?

It comes from Pip Al as a way to celebrate my daughter and n00bz's son, Pippa and Alexander. It also turns out to be the name of a type of fig and a village in Nepal.

Credits

I didn't realise it when I included them, but the "Hashcat", "Character sets" and "Character set ordering" stats are all based on an original idea by iPhelix in his tool PACK. If you are interested in generating Hashcat masks then his work is well worth a read.