Monthly Archives: March 2015

Monitoring Pastebin

Leave a reply

Suppose you want to monitor the Internet for certain keywords such as your username, email or some text pattern relevant for you or your company. A good starting point is to monitor Pastebin for those keywords, since that is where most of the interesting stuff gets dumped.

What I wanted was simple:

  • detect new pastes that match a given pattern
  • save matching pastes
  • do some processing on those pastes
  • notify me with the results

You can register alarms on Pastebin for up to 3 keywords and get notified upon matching, but you cannot use regular expressions or exclude keywords. Also, mail notifications are not the easiest to handle automatically.

The classical choice was to use Pastemon, maybe the first software of its kind. I had two problems with it: it would not save pastes to file and it was no longer maintained.
Searching for alternatives I found Pystemon, a newer and improved version of Pastemon.

So I ended up building the following setup:

1. setup Pystemon to download and store any pastes that match my rules.

  1. get (my version of) Pystemon git clone https://github.com/tmendo/pystemon.git
  2. edit pystemon.yaml
  3. set the archive dir
  4. enable and configure email alerts
  5. enable the use proxy option
  6. define some search patterns, for instance

    7.  - search: '[^a-zA-Z0-9]example.(com|org)'
   description: 'example domains'
 - search: 'tmendo'
   description: 'tmendo'
   exclude: 'tmendoza'

  7. run it with python pystemon.py -v -c pystemon.yaml

A few mistakes and Pastebin blocked my IP. Later they unblocked it and told me the threshold to stay safe, but in the meantime I have decided to proxy the requests through TOR. Since TOR only exposes a SOCKS proxy and Pystemon only supports HTTP proxies I used DeleGate.
Pystemon was originally designed to cycle through multiple proxies removing those that fail. I needed it to stick with the TOR proxy even if it eventually failed sometimes, so I forked the project, added support for a single proxy without removals, fixed some TODOs, included a few fixes from another committer, and did a pull request.

2. setup DeleGate to convert an SOCKS proxy to an HTTP proxy
Because TOR exposes a SOCKS proxy and Pystemon only supports HTTP proxy I am running DeleGate to pass requests between them.

  1. download the source
  2. tar zxvf delegate9.9.13.tar.gz
  3. make
  4. create dg.conf
    -P127.0.0.1:8080
-Tx
-fv
SOCKS=127.0.0.1:9050
DGROOT="/path/to/delegate/"
SERVER=http
REMITTABLE="http,https/443"
CACHE=no
TIMEOUT="shutout:30m"
MAXIMA="randstack:32"
MAXIMA="randenv:1024"
MAXIMA="randfd:32"
[email protected]

  5. execute with /path/to/delegated +=/path/to/delegate/dg.conf

3. setup TOR following these instructions.
This will install and start the daemon with the default configuration that generates new circuits every 10 minutes (you get a new IP every 10 minutes, at least). Pastebin does not discriminate TOR exit nodes, not sure about the other paste sites.

Test the DeleGate and TOR combination:

telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
host: mendo.pt

HTTP/1.1 301 Moved Permanently
DeleGate-Ver: 9.9.13 (delay=9)
Server: nginx
Date: Fri, 27 Mar 2015 14:38:04 GMT
Content-Type: text/html
Location: https://mendo.pt/
Strict-Transport-Security: max-age=31536000; includeSubDomains
Via: 1.1 - (DeleGate/9.9.13)
Connection: close
Content-Length: 178


profit!

4. setup a script that parses the stored pastes and do something useful.
You can use inotifywait to detect new pastes downloaded by Pystemon and do whatever you like with them (run it in a sandbox, do some further string matching, etc.)

#!/bin/bash

is_dir_pattern=\bISDIR\b

inotifywait -r -m /path/to/pystemon/output -e close_write |
    while read path action file; do
        if [[ ! "$action" =~ $is_dir_pattern ]]; then
            echo "The file '$file' appeared in directory '$path' via '$action'"
	    # process $path/$file
            # notify me of some finding
        fi
    done

EDIT
Listening to create and moved_to events might result in you being warned of a new file before its contents are written so you would end up with an empty file to process. Use close_write instead.

And thats it. Put everything under DJB Daemontools and you are able to monitor some paste sites for whatever you like and whatever reason.

Not even God forgives passwords in clear text

Leave a reply

I think this comic from CommitStrip says it all:
CommitStrip clear text passwords

Passwords stored in clear text are half of the recipe to scary headlines such as “thousands of accounts published online by hackers”. To complete the recipe just add a single SQL injection or a good path traversal/LFI mix.

Mature and trained developers rarely make such mistake, why? Because they had training and understand the consequences. The problem is the others, which need to be self-taught.

So let me be very clear:

  • you don’t need passwords in clear text to compare against the ones from login
  • you can compare representations of them, lets say an hash
  • never store or log passwords

Regarding the storing itself, there are way too many options [1]. I will recommend just one, which is adequate for most situations (if you feel this does not fits you, please contact the nearest security guy, or leave a comment):

function store(password)
    salt = secureRandom(32bits)
    hash = pbkdf2(salt, password, 10000)

    storeAtDB(salt;hash)
end

function validate(user, password)
    salt,hash = getUserFromDB(user).split(;)
    login_hash = pbkdf2(salt, password, 10000)

    if (login_hash XOR hash == 0 )
        return true
    else
        return false
end

I won’t explain these in detail, but the basic idea is to store an hash of the password plus a salt. No, it is not easy to find out the password given an hash + salt. No, it won’t slow down your application. Yes, you will be much, much safer.

[1] https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet