If you need to get all the information possible about numerous IP addresses, including all possible related emails, physical addresses, telephone numbers and more this post is for you.

I recently identified a very large number of critically vulnerable systems worldwide and I needed to contact the persons in charge of these systems to inform them of my findings and request their agreement to conduct further experiment on their systems.
The problem is, a simple whois, online or from the command line (e.g. “whois” in a Linux shell), will give numerous information about the host, but often the email is hidden for whois privacy protection.
After reading the manual of the command whois, I have seen that an extra parameter is available “whois -B” to deactivate the filters and give the email addresses.
It worked for European IPs (using the RIPE NCC RIR), but the command failed miserably for north American IPs (using ARIN RIR); for these IPs with this extra parameter, even the unprotected information was not given anymore.
However, I noticed that some specialised websites such as CentralOPs.net always return the emails when they are available, so I knew it was possible to retrieve them…

I searched for other platforms providing this feature, but all of them that I could find had many anti-automation measures from easily bypassable HTTP request analysis (checking the user agent, the Accept, Accept-Language, Accept-Encoding, and other fields in the headers) to more annoying email sent as an image, CAPTCHA systems and others.
I gave it a try, I will detail all this in a future post, but I decided to look for simpler solutions instead.

The other solution was to understand how these platforms have access to the privacy protected information.
These websites interrogate the authorities in charge of the IP addresses, the RIR (Regional Internet Registeries).
Wikipedia’s definition:

“A regional Internet registry (RIR) is an organization that manages the allocation and registration of Internet number resources within a particular region of the world. Internet number resources include IP addresses and autonomous system (AS) numbers.”

Followed by the 5 RIR in the world:

The Regional Internet Registry system evolved over time, eventually dividing the world into five RIRs:

African Network Information Center (AFRINIC)[1] for Africa
American Registry for Internet Numbers (ARIN)[2] for the United States, Canada, several parts of the Caribbean region, and Antarctica.
Asia-Pacific Network Information Centre (APNIC)[3] for Asia, Australia, New Zealand, and neighboring countries
Latin America and Caribbean Network Information Centre (LACNIC)[4] for Latin America and parts of the Caribbean region
Réseaux IP Européens Network Coordination Centre (RIPE NCC)[5] for Europe, Russia, the Middle East, and Central Asia

Each of these websites provide an online whois, sending the results in HTML, JSON, or XML.

So I made a Bash script that take in input a file containing 1 IP per line and then:

1. Run several whois commands with different parameters to be sure that none of them bug, saving the output to a separate file
2. Use WGET with the correct GET/POST information to retrieve the whois from the RIR online WHOIS, saving the output to other separate files
3. Merges all the result from all the whois methods into a big file
4. Use regular expressions to get all the emails from all the results (removing duplicate emails per host at the same time)

I share the script at the end of this post in case you want to use it.

The script can be improved by:

1. Check the return code of the whois requests (sometimes I get a connection refused or Timeout for some reason)
2. Add max retries and manual timeout to the wget commands
3. Spoof the user agent and other headers in the request to avoid behind an obvious script (Can be done with: –user-agent=”Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0″ –header=”Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8″ –header=”Accept-Language: en-US,en;q=0.5″)
4. Allow multiple requests to be performed in parallel (you can also divide your IP file and run different instances of the script in different directories)
5. You can run the script with the “nohup” (no hang up) command, so if your SSH stops or you close your terminal the script will keep executing
6. Perhaps my regular expression could be improved (it can include some false positives I think such as pierre@……..fr)

#!/bin/bash

ipFile="IPs.txt"
whoisDir="whois/"
logFile="log.log"
ripeSearchURL="https://apps.db.ripe.net/search/query.html?searchtext="
arinSearchURL="https://whois.arin.net/ui/query.do"
afrinicSearchURL="https://www.afrinic.net/index.php?option=com_whois&view=whois&Itemid=221"
apnicSearchURL="https://wq.apnic.net/whois-search/query?searchtext="
lacnicSearchURL="https://rdap.lacnic.net/rdap-web/ip?key="
suffixWhois="_whois"
suffixWhoisD="_whoisWithReverseDNSObj"
suffixWhoisB="_whoisUnfiltered"
suffixWhoisDB="_whoisUnfilteredWithReverseDNSObj"
suffixRIPEHtmlResult="_whoisRipeHtml"
suffixARINHtmlResult="_whoisArinHtml"
suffixAFRINICHtmlResult="_whoisAfrinicHtml"
suffixLACNICHtmlResult="_whoisLacnicHtml"
suffixAPNICJsonResult="_whoisApnicJson"
suffixMEGAwhois="_megaWhois"
suffixEmailsOnly="_emails"

if [ ! -d "$whoisDir" ]; then
    mkdir $whoisDir
fi

iteration=0
while IFS="" read -r currentIP || [[ -n "$currentIP" ]]; do
    iteration=$((iteration+1))
    currentIPtreatment=`echo $currentIP  | tr -d '\r' | tr -d '\n' | tr -d '\t'`
    currentIP="$currentIPtreatment"
    echo -n "[$iteration] $currentIP "
    
    # Saving whois and unfiltered whois command outputs and HTML result of online RIR DB
    whois $currentIP > $whoisDir$currentIP$suffixWhois
    echo -n "."
    whois -B $currentIP > $whoisDir$currentIP$suffixWhoisB
    echo -n "."
    whois -d $currentIP > $whoisDir$currentIP$suffixWhoisD
    echo -n "."
    whois -d -B $currentIP > $whoisDir$currentIP$suffixWhoisDB
    echo -n "."
    # RIPE answers with whois -B but I get also the web results
    wget -a $logFile -O $whoisDir$currentIP$suffixRIPEHtmlResult $ripeSearchURL$currentIP
    echo -n "."
    # ARIN doesn't give emails with whois -B, the web results will be our only source of emails
    wget -a $logFile -O $whoisDir$currentIP$suffixARINHtmlResult --post-data="queryinput=$currentIP" $arinSearchURL
    echo -n "."
    # AFRINIC answers with whois -B but I get it from the web interface anyway
    wget -a $logFile -O $whoisDir$currentIP$suffixAFRINICHtmlResult --post-data="key=$currentIP&action=get_search_result&ajax=true&source=AFRINIC" $afrinicSearchURL
    echo -n "."
    # APNIC
    wget -a $logFile -O $whoisDir$currentIP$suffixAPNICJsonResult $apnicSearchURL$currentIP
    echo -n "."
    # LACNIC
    wget -a $logFile -O $whoisDir$currentIP$suffixLACNICHtmlResult $lacnicSearchURL$currentIP
    echo -n "."
    
    # Creating a MEGA WHOIS file (all responses into one big file, for further use and convenience...)
    echo > $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixWhois >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixWhoisB >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixWhoisD >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixWhoisDB >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixRIPEHtmlResult >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixARINHtmlResult >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixAFRINICHtmlResult >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixAPNICJsonResult >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    cat $whoisDir$currentIP$suffixLACNICHtmlResult >> $whoisDir$currentIP$suffixMEGAwhois
    echo >> $whoisDir$currentIP$suffixMEGAwhois
    echo -n "."
    
    # Getting emails from the MEGA WHOIS file
    grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" $whoisDir$currentIP$suffixMEGAwhois | sort | uniq > $whoisDir$currentIP$suffixEmailsOnly
    echo " DONE!"
    
done < $ipFile