How to serve files with a simple and quick PHP BitTorrent tracker

I needed to share some raw video files with a friend lately so I uploaded 17GB of data to my webserver and sent them the HTTP link. Problem was the video files were scattered over several directories and the directory structure was important so the end user would have had to download each one individually if they didn’t have some sort of download manager to download all. I could have given them anonymous FTP access but the end user isn’t that technical so I had to keep it as simple as possible. I could have zipped up the files on the server but that was taking ages for 17GB and would be a disaster if somehow the big ZIP file corrupted.

So I decided to make a torrent of the data. The great thing about torrents is nearly everybody knows how to use them no matter how technophobic they are as they all want to download Game of Thrones every Monday morning. I already have rtorrent and mktorrent on my Linux server so I already had a way to seed and a way to create the actual torrent files. I was just missing a tracker. I wanted the quickest and simplest option, and I found it in Bitstorm via an article on TorrentFreak.

You can download BitStorm’s one file of PHP source here.

1) Save as ui.php and upload to your web server to a publicly accessible folder. So your URL should be something like http://www.georgiecasey.com:80/ui.php
Notice the port number of 80, the standard HTTP port. You need to specify this in your torrent files as a BitTorrent tracker can be on any port and won’t default to 80.
2) Change permissions so script has write access to /dev/shm/ to track peers. So chmod 0755 ui.php or something like that.
3) Create the torrent file of the folders you want to transfer with mktorrent and the announce URL of your ui.php file. eg mktorrent --announce=http://www.georgiecasey.com:80/ui.php folder_of_files/
4) Start a rtorrent instance where your folder_of_files is and add your new torrent. It should do a hash check and start seeding. It’s best to open rtorrent in a screen so you can leave it running.
5) Send the torrent to whoever you want to download the files.

Damn, it feels weird using BitTorrent in legal ways!

How to disable Clickberry appearing on your Youtube videos

Like me, have you suddenly started seeing a ‘tag’ feature on your Youtube videos, that when clicked show a ‘Share moment’ and ‘Share object’ option and a little icon of a berry appears on your Youtube play bar? And when you click this berry icon, it brings you to clickberry.tv? Looks like this:

Unclicked

Unclicked

When Clicked

When Clicked

I thought the tag thing was a new feature of Youtube along with the Google+ comments but it’s actually added by a Chrome extension gone rogue called ‘FVD Video Downloader‘. You might remember that extension asking for more permissions in the last few days. Just uninstall and Clickberry will be gone from your Youtube.

How to use Overplay and other VPNs as a cURL proxy

Proxies are like hard drive space, you can never have enough. Or enough IPs to be more accurate as Facebook, Google and other services are getting better at flagging the IPs of popular HTTP/SOCKS5 proxies. So we need to find fresh proxies to use in our PHP/Python scripts. A great source is the proliferation of VPN services that are popping up as consumers worry more about their Internet privacy. Problem is these are intended for use by endusers on their desktops and not in serverside PHP scripts. So it’s a bit tricky to get these working with cURL, but fear not, I explain all in this post.

The magic of being able to use a VPN in cURL is the CURLOPT_INTERFACE option. This lets you set the network interface that cURL uses. You can’t use a VPN directly in cURL as cURL/PHP operates on a higher network level than the VPN protocol.

1)
So we need to setup the VPN on a new interface. Note that you absolutely need root access to your server to create interfaces so this guide is only useful for people with their own dedicated servers. People on shared $2/month servers are shit out of luck. You might get it working on a VPS, I’ve no idea. So to create an interface, you need to download and install OpenVPN if you haven’t it installed already. There’s loads of info online to help you do this, so figure it out and come back.

2)
Next we need to get the configuration files we need from our VPN provider of choice. I use Overplay.net. You need an account with them and it cost something like $5 a month. Download the ZIP file of connection files and unzip on your server in a new directory. Also download the Overplay public key certificate and make sure it’s in the same directory:
curl --insecure -o connections.zip https://patch.oppatch.com/connections.zip
unzip -u connections.zip
curl --insecure -o OverplayCert.crt https://patch.oppatch.com/OverplayCert.crt

Now we have the connection files which work fine if you run Linux as your desktop OS and just want to browse the web as described in this Overplay guide. But we DO NOT want to just start the VPNs as is as it will take over the main Internet connection and make your server inaccesible. I did this a few times and had to get my host to reboot my server.

3)
So we need to edit the configuration files and add one command, route-nopull. This prevents Overplay from taking over the routing information. If you take one thing away from this blogpost, it’s the addition of the route-nopull option as it’s what lets you use these config files on your server.

I also want to add my login as a file so I don’t have type it everytime. So create a new file in the same directory and name it ‘auth_overplay’ or whatever you want. Enter your username and password, seperated by a newline. So if we’re taking the ‘Overplay – Ireland-1.conf’ file as our example, our config would now look like this. Our additions are in bold at the bottom:
client
dev tun
proto udp
remote 176.67.85.4 1443

resolv-retry infinite
nobind
persist-key
persist-tun
ca OverplayCert.crt
verb 5
route-method exe
route-delay 2

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
comp-lzo

route-nopull
auth-user-pass auth_overplay
#daemon

Note we also added daemon at the end, but commented it out. You can uncomment this when you’ve got everything working and want to start the VPN as a daemon so you can use it without having to have the SSH window open.

4)
Now start up the VPN with OpenVPN:
openvpn "Overplay - Ireland-1.conf"

If everything works, you should see output like this:
Fri Jul 26 21:05:13 2013 us=31236 OpenVPN 2.1.3 x86_64-pc-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [MH] [PF_INET6] [eurephia] built on Feb 21 2012
Fri Jul 26 21:05:13 2013 us=31325 WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Fri Jul 26 21:05:13 2013 us=31331 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Fri Jul 26 21:05:13 2013 us=31678 LZO compression initialized
Fri Jul 26 21:05:13 2013 us=31725 Control Channel MTU parms [ L:1574 D:138 EF:38 EB:0 ET:0 EL:0 ]
Fri Jul 26 21:05:13 2013 us=31748 Socket Buffers: R=[124928->131072] S=[124928->131072]
Fri Jul 26 21:05:13 2013 us=31766 Data Channel MTU parms [ L:1574 D:1450 EF:42 EB:135 ET:32 EL:0 AF:3/1 ]
Fri Jul 26 21:05:13 2013 us=31777 Local Options String: 'V4,dev-type tun,link-mtu 1574,tun-mtu 1532,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-client'
Fri Jul 26 21:05:13 2013 us=31781 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1574,tun-mtu 1532,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
Fri Jul 26 21:05:13 2013 us=31794 Local Options hash (VER=V4): 'd3a7571a'
Fri Jul 26 21:05:13 2013 us=31802 Expected Remote Options hash (VER=V4): '5b1533a2'
Fri Jul 26 21:05:13 2013 us=31811 UDPv4 link local: [undef]
Fri Jul 26 21:05:13 2013 us=31816 UDPv4 link remote: [AF_INET]176.67.85.4:1443
WRFri Jul 26 21:05:13 2013 us=37221 TLS: Initial packet from [AF_INET]176.67.85.4:1443, sid=a552afa0 928c908a
WFri Jul 26 21:05:13 2013 us=37266 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
WRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRFri Jul 26 21:05:13 2013 us=72079 VERIFY OK: depth=1, /C=UK/ST=LANCS/L=MANCHESTER/O=OVERPLAY.NET_LLP/OU=CA/CN=OVERPLAY_CA/emailAddress=ca@overplay.net
Fri Jul 26 21:05:13 2013 us=72206 VERIFY OK: depth=0, /C=US/ST=IL/L=Chicago/O=OVERPLAY.NET_LLP/OU=SERVERS/CN=vpn1-us/emailAddress=ca@overplay.net
WRWRWRWRWWRRWWWWRRRRWRWRFri Jul 26 21:05:13 2013 us=472472 Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Fri Jul 26 21:05:13 2013 us=472487 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Jul 26 21:05:13 2013 us=472528 Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Fri Jul 26 21:05:13 2013 us=472534 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
WFri Jul 26 21:05:13 2013 us=472560 Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
Fri Jul 26 21:05:13 2013 us=472576 [vpn1-us] Peer Connection Initiated with [AF_INET]176.67.85.4:1443
Fri Jul 26 21:05:15 2013 us=598837 SENT CONTROL [vpn1-us]: 'PUSH_REQUEST' (status=1)
WRRWRWRFri Jul 26 21:05:15 2013 us=604235 PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 176.67.84.100,route 10.204.0.1,topology net30,ping 10,ping-restart 120,ifconfig 10.204.0.54 10.204.0.53'
Fri Jul 26 21:05:15 2013 us=604257 Options error: option 'redirect-gateway' cannot be used in this context
Fri Jul 26 21:05:15 2013 us=604275 Options error: option 'route' cannot be used in this context
Fri Jul 26 21:05:15 2013 us=604293 OPTIONS IMPORT: timers and/or timeouts modified
Fri Jul 26 21:05:15 2013 us=604297 OPTIONS IMPORT: --ifconfig/up options modified
Fri Jul 26 21:05:15 2013 us=604301 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Fri Jul 26 21:05:15 2013 us=604472 TUN/TAP device tun0 opened
Fri Jul 26 21:05:15 2013 us=604485 TUN/TAP TX queue length set to 100
Fri Jul 26 21:05:15 2013 us=604507 /sbin/ifconfig tun0 10.204.0.54 pointopoint 10.204.0.53 mtu 1500
WFri Jul 26 21:05:17 2013 us=671302 Initialization Sequence Completed

Now open a new SSH session as root and enter ifconfig to see the list of network interfaces. You should see your proxy listed with the interface name tun0 or something like that.

Some OverPlay servers don’t complete the TLS handshake for me but I’m thinking that was because they were old IPs. Overplay seems to change actual proxy servers a lot.

5)
If you see the tun0 interface in ifconfig, then it worked, probably. Test it with cURL on the command line:
curl --interface tun0 http://icanhazip.com

Replace tun0 with whatever your interface is called. icanhazip.com is a simple site that outputs your IP and nothing else and has been online for 3 years so hopefully it stays online. If everything works, the site should output the IP of Overplays proxy and not your server IP.

To use the interface in your PHP scripts, you’d set it with something like this:
curl_setopt($curlh, CURLOPT_INTERFACE, "tun0");

I found from experience that PHP/cURL defaults to using the standard network interface if something doesn’t work, so it grabs pages using your real IP! This can be a disaster if you’re doing dodgy stuff and don’t want to get banned from FB or Google. For this reason, I always grab icanhazip.com in my scripts and check to make sure it’s not my server IP.

Well that’s about it. I also wrote a few PHP scripts that automatically download the Overplay config files, edit them, check what country the IP is and add them to a MySQL table. Then I’ve other PHP code that just queries the table grabbing a random proxy from the country I need. I’ll do a blogpost and share that code if anyone actually reads this one and comments. And also comment if you run into any trouble, I’m usually quick to answer.

iMessage is a handy way to show people you’ve the money for an iPhone

ryanair leaving money on the table with no customer registration

I’ve booked loads of Ryanair flights and every time they ask you for your name, address, mobile and when you check in online, they ask for your passport number and date of birth. Why do you have to do this everytime? Why can’t you register an account on the ryanair website and have all this information stored for you so you can book flights with one click. It would make life easier for customers and provid loads more potential cash for Ryanair.

They could start a self-serve ad system like Facebook’s and let advertisers target customers by nationality, destination and age. Right now they just have generic Hertz ads when you book your flight which I doubt bring in much conversions. How much would a small car hire firm in Knock make if they could target all German people over the age of 40 with a German language ad right after they’ve booked their flight to Knock? And they more targeted and higher converting ads, the more money for Ryanair. And this is guaranteed correct info, nobody is going to put a fake nationality or date of birth on their online check in.

Of course all this data gathering would need a pretty good terms and conditions but I sure that could be drawn up easy. Ryanair are leaving so much €€€ on the table, their IT department must be retarded.

charles web proxy review

As I said in my last post, I’ve written many a scraper using php with curl or fsockopen in my time, trying to write automated tools and scraping data. I’ve tried many tools to help me sniff the HTTP traffic so I could emulate it in PHP as quick as possible. I started off using Wireshark or Ethereal as it was called at the time which was complete overkill, mostly used for network trouble shooting and grabs all TCP/UDP packets which is information overload, all we want is HTTP data. Then I think I used the LiveHTTPheaders addon for Firefox which was pretty limited. Then a Java program called Burpsuite which was pretty powerful but I ran into a problem trying to automate myspace myads submissions, trying to figure out what HTTP the myads flash file was sending over HTTPS. I ran the gamut of every proxy tool out there until I came across Charles Web Proxy.

It’s basically the best out there. It sits as a proxy between the web and your browser, grabbing all data as it comes in. This usually causes problems with SSL but it has a custom SSL cert that you manually add to your browser that lets you log HTTPS data with no warnings. It can grab Flash traffic as it seems to work as a Windows proxy, not just a browser one. It presents HTTP data many different ways so you can understand what’s going on quicker. For example, a multipart form upload is presented as the the raw HTTP data sent, just the headers, just the cookies, the text body and all the form fields. I won’t list all the features as they’re all listed on the site. If you’re using any other tool for automation/scraping, you’re wasting time.

php curl debugging – seeing the exact http request headers sent by curl

In my many of years of php/curl use, I’ve hammered my head off my table countless times trying to debug scripts that weren’t emulating the browser like it was supposed to. This was pretty hard without seeing the exact HTTP request header sent by cURL each session, but this is possible now from PHP 5.1.3

Use the curl_getinfo php function with the CURLINFO_HEADER_OUT option but make sure to set option CURLINFO_HEADER_OUT to true as a curl option.

$ch = curl_init("http://www.google.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
$get = curl_exec($ch);
$info=curl_getinfo($ch,CURLINFO_HEADER_OUT);
var_dump($info);

how to copy a website with httrack on linux

This is more for my own reference than anything. Say you see a flog on the intertubes and want to rip it and stick up for affiliate links. How to do it quickly on Linux? I used to use wget but it sucked. httrack is much better.

httrack "http://www.techcrunh.com/" -N1 -O "/home/techcrunch_rip/public_html" +techcrunch.com/* +crunchgear.com/* -v

This will rip the homepage of techcrunch and stick it in the folder specified by -O. URL filters next ensure it only downloads files from certain domains. The -N1 argument is the most important, it ensures htttrack sticks all images, css in one directory instead of creating loads of directories. Very handy.

irish phonebook on your iphone

I was looking for an iPhone app a while ago to search Irish business phonenumbers and couldn’t find one, so wrote one myself. And while I was waiting for Apple to approve my app, a different phonebook app was released with better user interface! BUT it just searches the goldenpages website so you need a net connection. I scraped the Goldenpages website and stuck it in the app, so no net connectio needed, handy when you quickly need a number.

Check it out here, only e2.39 to buy