GIVE Engineering

15 Aug 2012

This is a book about the technology and the engineering discipline at GIVE. As I will move to Twitter very soon (on Oct 2012; right now it is Aug 2012), I need to pass all of my technical knowledge to the next wave of GIVE engineers.

Supreme Law

Things change rapidly

Any work that you have done; you might need to throw it away, change it, or completely re-write it. There is no way out.

GIVE explores the frontier of giving. We don't exactly know what works and what doesn't work; we only know it roughly. It's the reality you have to live with.

There is no other way unless you shift your attitude.

Be forgiving to the management, and be willing to change things. (Believe me, the management also tries very hard to plan it right at the first try)

Here is a wisdom from Paul Graham:

You'll be better off if you operate like Columbus and just head in a general westerly direction. Don't try to construct the future like a building, because your current blueprint is almost certainly mistaken. Start with something you know works, and when you expand, expand westward.

Never work on anything that takes more than a week

In other words, never tell the management that it takes 2 (or more) weeks to finish a certain task.

Instead, you should blatantly reject the task and ask the management to reduce it to a simpler but working version of the task. And, of course, we increment it later.

Try hard to automate and embed everything

Automating and embedding everything accelerates the development process. It facilitates the process of adding a programmer to the team, replicating project on another machine, TBD..

For example, write a bash script for every maintainence-related (e.g. backup database) task because you are very likely to forget the command and arguments later in time.

Another example is where a binary of MongoDB is checked in with Git. It is possible because it is only 8mb… and it is so convenient for many use cases.

Server Installation on EC2

Server Access

Within the credential ZIP, there is a file `giveasia.sh. You can ``./giveasia.sh` in order to access the server as the userec2-user``.

You can `sudo su rootin order to switch to the user ``root```.

Secure Rails

Policy

  • The credentials of third-party accounts shall never be stored in the project, and they MUST NOT be checked into Git
  • We create a new deploy user for each engineer, and deactivate the user when the engineer leaves the company
  • All the credentials are stored in a protected ZIP file, which is in `admin@giveasia.org`'s inbox

We have 3 groups in order to secure our Rails. Here are they:

  • production for handing the production server
  • staging for handing the production server
  • nginx for handing Nginx

Setup

After deploying, we symlinks all the files in `/mnt/ebs/giviki2_credentials/*to ``/mnt/ebs/giviki2/current/config/*` with this commandcp -sf /mnt/ebs/giviki2_credentials/* /mnt/ebs/giviki2/current/config``

We need to set proper permissions for all relevant folders:

  1. Create groups: `productionand ``staging```
  2. Create users: `nginx-production, ``nginx-staging`,deploy-production``, and `deploy-staging`
  3. Assign `nginx-productionand ``deploy-production` toproduction``
  4. Assign `nginx-stagingand ``deploy-staging` tostaging``
  5. Create user `nginxand assign it to ``production`,staging``, and `nginx`
  6. Change the folder `/opt/nginxto belong to ``nginx```
  7. Add SSH key for deploy-production and deploy-staging
  8. Modify sudoer in order to allow `deploy-*` to run rake tasks

We allow deploy-production to call /usr/local/bin/bundle exec in the name of nginx-production

and deploy-staging to call /usr/local/bin/bundle exec in the name of nginx-staging.

Because deploy-production cannot read credentials. For deploy-staging, we merely want to make them analogous

Here is how we do it with `visudo`:

Defaults env_keep += "RAILS_ENV" Defaults:deploy-production !requiretty Defaults:deploy-staging !requiretty Cmnd_Alias RAILS_CMDS = /usr/local/bin/bundle deploy-production ALL =(nginx-production) NOPASSWD: RAILS_CMDS deploy-staging ALL =(nginx-staging) NOPASSWD: RAILS_CMDS

Lesson learned: the command must start with / (implied a full path)

We can call `sudo -u nginx-production /usr/local/bin/bundle somethingfrom ``deploy-production```

A manager will have his/her SSH key added to deploy-production. An engineer will have his/her SSH key added to deploy-staging.

Disadvantage

Here are the insidious things Coder can do:

  • Steal credentials by showing them on a web page
  • Redirect payment to Coder's Paypal

These are the 2 examples of why Manager has to review the code, and only the manager can deploy to production.

It's a tedious task; well, suck it up, until I can find a better solution.

Writing a script

When you are writing a script, please keep in mind the below principles:

Resumibility

Ok, I don't know which word I should use… A script should be able to resume at any point of time; it won't perform redundant work, if it shouldn't.

Please be aware that a script is usually a long-lived task, therefore, it will be interrupted or corrupted while running.

The thing I normally do is to create a transaction model to ensure that what was processed is not being processed again.

Know the difference between database migration, Rake, and Runner script_

  • Database migration is for alter the structure of the database
  • Rake is for tasks that will be used many times
  • Runner script is for one-time task

Scaling

We don't care about performance right now.

What I've learned

Don't use `has_many` Mongoid

Use `belongs_toinstead. With ``has_many```, Mongoid stupidly stores an array of ownees' ids within the owner's record.

Give it a kudos