Category Archives: HowTo

How do I _____?

Dynamically generating data types in python

For a project I was working on recently, I needed to define a bunch of data types in python. As defining 10 different datatypes with about the same functionality manually would’ve been a pain, I decided to try something else.

I already had a bunch of nested dictionaries defining the fields and types of all datatypes, as I was generating SQLite statements from them. The general format looked something like this:

datatypes = {
    "type1": {
        "field1": {
            "type": "INTEGER",
            "notNull": True,
            "primaryKey": False,
            "autoIncrement": False,
            "default": None,
            "foreignKey": {
                "table": "type2",
                "field": "field1",
                "onDel": "RESTRICT",
                "onUpd": "RESTRICT"
            }
        },
        "field2": {
            # ...
        },
        # ...
    },
    # ...
}

Each outer dictionary (“type1”) defines a data type, and each inner dictionary (“field1”) defines a key-value-pair of this data type, including some information like “can this be empty?”. So, I already had everything I needed to define my data types.

What are the data types supposed to be able to do? I needed them to have unchangeable values (meaning that I needed only “getters”, no “setters” outside the constructor). So, how can I create class definitions from this block of definitions?

Easy. I generate a long string containing all definitions I need and run it through exec, a function I usually tend to avoid like the plague (note that this assumes that the class definition is safe and cannot be changed by others, meaning that you don’t have to worry about the security ramifications of using exec).

The code for the generation itself only clocks in at 55 lines, including comments. For easier reading, I put it in a gist.

As you can see, the code will generate a bunch of definitions for each data type, including getters and a checkRep-Function that can be used to verify the consistency of the data. So far, it does not check for foreign key constraints, as they are quite annoying to check and enforced in the database backend anyway.

So, why is this awesome?

  • You don’t have to manually write out all your datatypes. Instead, you define them once and then generate them automatically.
  • Even large changes to datatypes are quick and easy, by just changing the definition in one place.
  • You can generate other things like DB interfaces and -definitions as well, all from the same source definition.

And what are the problems with this approach?

  • You are using exec, meaning that if someone untrusted gains access to your definitions, they can do evil things to your code.
  • code coverage tests don’t play well with it.

For a full example of the code in use, you can check out my (abandoned) InvoiceManager-Project on GitHub. There, I also generate database definitions, database validation code, database interfaces, and unittests for all of these things, all from the same source definition.

Let me know what you think.

Howto: Usable disk encryption with Linux Mint 16

After I encountered a bunch of problems when setting up full disk encryption on Linux Mint 16, I thought I’d share the final solution I chose, and how to avoid some of the bugs in the Linux Mint Installer.

This howto is based on the Cinnamon variant of Linux Mint 16 x64, but it is probably applicable to other variants and architectures of Linux Mint 16.

Now, there are a few ways to partition your hard drive when using Linux. I wanted the following setup:

  • /boot, unencrypted, 256 MB (otherwise the system won’t boot)
  • /, encrypted, ~50 GB
  • swap-space, encrypted, ~10 GB
  • /home, encrypted, the remaining ~440 GB

This setup has the advantage that you can preserve the /home-partition when updating the system, saving you the trouble of making a backup and restoring it afterwards.

Now, the first option you have when installing Linux Mint 16 is to just select “encrypt this installation” during the install process. Sadly, this creates a setup without a dedicated /home partition, so it is out of the question.

Another Howto I found proposed to use the partition manager of the installer (select “something else” when asked how you want to install Linux Mint 16), and create the three encrypted partitions with the “Use as: Physical volume for encryption” option. This works, but it will make you enter your decryption passphrase three times in a row when booting up your PC, which is pretty darn annoying, so I went looking for a better option.

The way I finally solved my problem was like this: When installing Linux Mint 16, go to the partition manager and create the following setup:

  • /boot, unencrypted ext2, 256 MB
  • An encrypted partition with the size you want for the root of the file system (“/”, choose “Physical volume for encryption”, and a new virtual hard drive will appear, where you will have to click on the only partition on it and select “Use as: ext4 journaling file system”, setting the mount point to “/”)
  • A partition with the size you want your swap space to have, but choosing “Do not use this partition” in the “Use as”-Dropdown menu (more on this later)
  • An unencrypted partition with the size you want for your /home, choosing “ext4 journaling file system” with a mount point of “/home”

Now, there are a few things strange about this setup:

  • We do not activate the swap space because Linux won’t let you install a system with an encrypted partition and unencrypted swap space, and we will encrypt the swap later.
  • We do not encrypt the /home partition because that will happen later as well

Now, after you’ve set up your partition table to your liking (substitute ext4 for other file system types if you want to, but I’ll stick to ext4 for now), click “Install now”. The setup will warn you about your system not having any swap space, but you can ignore that (as long as you have enough RAM to install and boot linux without swap, otherwise you’re out of luck with this howto).

Proceed with the installation until it asks you to create your user. Enter all your user information and do not check the “encrypt my files” option. Why? Because the installer repeatedly failed when I did it, and the one time it worked it produced an unencrypted home directory regardless of my settings. Continue the install and reboot into your fresh linux when the installation is finished. Make sure you are asked for the passphrase you set up for the root partition when booting!

Now, after booting into your fresh linux, make sure “ecryptfs-utils” is installed (sudo apt-get install ecryptfs-utils), and install “gparted” (sudo apt-get install gparted). Then, go to the “Users and Groups” setting and create a new administrator user. Set a passwort for it, then log out and back into the new user. Start “gparted” and select the unused partition that was intended as swap space (make sure it is the correct one, as the encrypted “/” partition looks very similar. Compare sizes to be sure). Rightclick it, and select “format to => swap”. Apply your changes, then right-click the newly created swap space and select “swapon”.

Next, open a terminal and enter sudo ecryptfs-migrate-home -u USER, where USER is the username of your primary user (not the one you are currently logged in as). Follow the on-screen instructions. After the migration finished, run sudo ecryptfs-setup-swap and follow the instructions. Once you are finished with this, log out (but do not reboot) and back into your primary account. Open a terminal and enter ecryptfs-unwrap-passphrase and note the passphrase down somewhere secure. This is your emergency passphrase for when you need to decrypt your home directory manually and / or have forgotten your user password. Afterwards, delete the user you created and reboot for good measure.

You should now have the following setup:

  • /boot, unencrypted
  • /, encrypted
  • /home, unencrypted, but containing an encrypted /home/USER.
  • swap-space, encrypted

You can verify this by running lsblk and checking if the relevant partitions show up as encrypted or not, and running df and looking for /home/USER/.Private in the leftmost column. If it is there, you should be safe.

This setup leaves you with encryption for all your relevant data, with minimal additional annoyance compared to an unencrypted linux (you only need to enter one passphrase on boot, your home directory will be decrypted on login). Now, a few caveats:

  • In this setup, only the home directory of the first user is encrypted. If you ever create additional users, you will have to repeat the steps ecryptfs-migrate-home and ecryptfs-unwrap-passphrase for the new user, or it will not be encrypted. (Technically, this means that this is no full disk encryption, but as long as you remember to do this for every new user or just don’t create additional users, you should be just as safe).
  • Your home directory can be decrypted with your users login passphrase, so choose a strong user password.
  • Encrypting the swap space in this fashion will break Linux’ hibernate function, so don’t use that afterwards (standby is fine, although that will leave everything decrypted in case your laptop is seized while it is in standby, so you may want to avoid that).
  • I have not tried reinstalling linux with an encrypted home directory yet, but according to this askUbuntu-Question, it should not be a problem.

I hope I could help you living a more encrypted life. Please let me know if you find any mistakes in this guide or if something is not clear from what I’ve written.

Update: There is a persistent bug in Ubuntu / Linux Mint, which leads to the SWAP partition not being mounted on reboot. A workaround is described here. Thanks to Igo in the comments for testing the workaround and reporting back.

Howto: Running Tor on a Synology DiskStation

Note: All of these steps may no longer be necessary. Check out this comment for a software package for your DiskStation, if you trust a version of Tor you have not compiled yourself.

(Repost from my tumblr)

After a brief conversation with the Tor support, I tried to and suceeded at getting Tor to run on my Synology DiskStation 211j. I suppose the setup process will be similar on all DiskStations and possibly other BusyBox NAS Systems, but I only tried my own one.

I suppose you already know your way around your NAS, in having SSH enabled and secured (important!) , and ipkg installed. You should also know basic stuff about linux (editing files, creading directories, sudo / su, …), but you don’t need to be an expert (hell, I am mostly a newbie myself when it comes to Linux).

Please also be aware of the legal implications that come with running Tor. I am not responsible for anything that happens to you, your NAS, your Network, Internet Connection, computer, data, cat, or anything else. Also, please note that while the following steps have worked for me, they might not work for you, and chances are that I will be unable to assist you in any way. Use Google or whatever search engine you are comfortable with to find solutions.

A note on the ipkg version of Tor:

I have asked the guys at TOR, and the version on IPKG is not official. It is also outdated, so please don’t use it. Compile TOR yourself instead.

Step one: Getting the Tor Source code

There are, as of April 2012, no precompiled ARM binaries available, so you will have to compile Tor yourself.

Go to https://www.torproject.org/download/download.html.en and download the Source Tarball (That’s important. Do not download any precompiled linux package).

Copy it to your NAS in some way (via a network share, for example). Getting the source Tarball directly on the NAS was not possible for me, as it is only loadable via https, and my wget had no https support compiled in.

Copy it to a location of your choice (your home folder, for example), and unpack it using:

tar x -f name_of_source_tarball.tar.gz

(remember you can autocomplete the filename with tab)

Step two: Checking the config

This step is easy. Just run “./configure” from the unpacked directory (you may have to “cd” into it first).

You will most likely get errors. Don’t freak out, thats normal.

If it complains that you don’t have gcc installed, just run “sudo ipkg install gcc” and you should be fine.

Usually, it will tell you that it has found a shared library, but is unable to use it, and you can specify a new path using the “—with-[libraryname]-dir=path/to/library” switch.

Most of the libraries will be located at /opt/lib

If you are indeed missing a library completely, you can most likely install it using ipkg.

For example, I was missing the openssl-libraries. By running “sudo ipkg list | grep openssl”, I was able to locate the “openssl-dev”-package that contained the libraries. If you really can’t find the libraries, use a search engine to figure out how to get them.

Once you get the “./configure” command to run without errors, using the switches explained above, you can run “make” (or install it first, if you don’t have it already, using “sudo ipkg install make”).

This will take a while (about 10 minutes for me). It should run without errors. If you encounter problems here, I will most likely not be able to help you, so use your friendly search engine again.

Step three: Preparing the system

First off, if you have not properly sealed your ssh, now is the time to do it. Use keyfiles to log in, change the standard port, disallow root login and so on. I will not go into details here, there are enough tutorials for that online.

Make sure all your software is up to date (“sudo ipkg update”, followed by “sudo ipkg upgrade”), and that your router’s Firewall is blocking every port by default. Be a bit paranoid.

If you are done with that, run “sudo mkdir /root/.tor”, followed by “sudo chown -R [your_username] /root/.tor”. This will enable Tor to use the directory, as per standard config.

Alternatively, you could just run “sudo [path_to_tor_source_dir]/src/or/tor” and then, after a second, cancel the execution using ctrl+c. Tor should create all required directories. Now you can run it without the “sudo” to get a list of all relevant directories that were created (Because it will complain that it has no access to them). “chown -R” all of them to your user, as described above.

Step four: Preparing a torrc file

Torshould have created a folder called “tor” somewhere (For me, it was /opt/etc/tor). cd to that folder and edit the torrc.sample (Or maybe it will be called torrc, without the .sample).

Read through it carefully and consider your choices, then make your changes. Also, check if you have write access by changing something and saving. If it works, keep going. Otherwise, exit your editor and restart it using “sudo”.

Keep in mind that the “#” character is signifying a comment. So, make sure the relevant lines are uncommented.

The most important decisions you have to make are:

  • SocksPort: Set to 0 if you only want to run a relay / exit node
  • Log configuration: It is useful to set a logfile for “notice” level logs. For example: “Log notice file /path/to/the/file/filename.txt”
  • RunAsDaemon: If you want Tor to keep running in background if you terminate your SSH connection, set this to 1. In this case, it is important to set a log file, or you will be unable to find out what is going on inside Tor, if there are any problems.
  • Port: Set some some port and make sure that it is forwarded in your router.Only set this if you want to run a relay, bridge, or exit node.
  • NickName: Set anything here to identify your Node. Again, only set if you want to run a relay, bridge or exit node.
  • RelayBandwidthRate: Set if you want to limit traffic through your relay, bridge or exit node.
  • RelayBandwidthBurst: Same here
  • AccountingMax, AccountingStart: Same here
  • ContactInfo: Set if you want the Tor team to be able to contact you, should something be wrong. Search engines are indexing this, so spammers will find your email eventually, if you are setting this.
  • DirPort: If you want to mirror directory information, set this and make sure your Router forwards the port.
  • DirPortFrontPage: Specify a HTML document that should be displayed if someone browses to your IP on your DirPort. Totally optional
  • MyFamily: Set the fingerprints of other Tor relays you are running here.
  • ExitPolicy:This is critical!If you want to only relay traffic (From Tor into Tor, as opposed to from Tor into the Internet), set this to “reject *:*”. Else, you can reject special ports, for example BitTorrent, Usenet, …If you chooseto be an exit node, you will get problems at some point, because people are using Tor to do illegal stuff, and your IP will show up eventually. Consider this carefully. Running a non-exit relay is safe and very much helps the Tor network.
  • BridgeRelay: If you want to serve as a bridge, set this to 1.

If you need your fingerprint to configure other relays, check “/root/.tor”.

Don’t forget to remove the “.sample” from the torrc file, if it was still there (“mv torrc.sample torrc)

Step five: Running Tor for the first time (for real)

Run TOR using “[path_to_Tor_sources]/src/or/tor”. If the torrc file is not in the standard directory, you can specify the path using “-f /path/to/file/torrc”.

If you have set Daemon to 1, check your log files. It should say “Self-testing indicates your ORPort is reachable from the outside”, as well as “self-testing indicates your DirPort is reachable from the outside” (If you have enabled the respective ports). If there are problems, check your port forwarding, paths and permissions.

Step six: Killing Tor if is is a daemon

If you have set Daemon to 1 and need to kill Tor for some reason, run “ps | grep tor”, note the PID of the tor process and run “sudo kill -SIGINT [tor_pid]”. It will take 30 seconds for Tor to shut down. If you need to shut it down fast, without regard for the stability of the currently connected clients, you can just use the kill command without the -SIGINT. Please try to avoid this.

That’s it. You are now (hopefully) running a Tor relay, or at least have access to Tor, using your NAS as a Proxy.

Some notes:

  • Subscribe to the tor-announce mailing list if you want to be notified on new Tor updates. Install them ASAP, as old versions might be insecure.
  • IMPORTANT: Read this page on the Tor documentation about improving security of your Tor Relay. There are many things you can do to make it harder for people to break into your machine. This is especially important if you are running an exit node.
  • If you want to browse the internet using Tor, use the Tor browser bundle instead of setting Tor as a proxy in your Browser. The TBB contains a hardened version of Firefox that has some additional tracking and exploit protections built in that your regular browser does not have. Do not expect to be anonymous if you use your regular browser with a Tor proxy.

So, that’s it, this time for real. If you have any notes concerning the process, do not hesitate to comment.