T-Mobile PAYG Mobile Broadband USB stick is available at Argos online and instore for £9.99 (out of stock online, but South London stores have a few instock). It was originally listed as £19.99 in the catalogue. You also aren't forced to buy a topup like you do with most other cheap PAYG phones and dongles.
If you want to use this a lot then this isn't that great a deal, but if like me you only want to use it once or twice a month this is perfect. You can topup and pay just £2 per day for unlimited usage, and your credit will only expire after 6 months of no usage. They also do £5 for a week and £15 for a month, but you can probably find better deals elsewhere if you want to use it that much. To the naysayers saying this isn't a good deal, please reread this paragraph and understand my use case.
The device itself is a Huawei E173, most of the Huawei devices are easily unlockable so I expect this is too. If anyone has any links to guides it would be appreciated. It takes a standard SIM card, but you will want to get a data SIM otherwise you'll most likely be charged extortionate prices.