We have a sharded architecture, which means data is scattered across several “shards”. Each shard has different data than all others. Each shard is a master-master pair. MM pairs are masters and slaves at the same time. They not only give you fault tolerance, they divide the read and write load between them that’s impossible to do in the common master-slave(s) setup. MM pairs have their own set of challenges.

Don’t Let Your Database Generate Anything

The main problem with MM pairs is caused by non-deterministic values generated by the database engine itself, such as autoincrement fields, random numbers and timestamps. The solution to that is that we don’t let the db generate anything. Every value inserted/modified always comes from the application. This allows us to write to either of the two sides of a MM pair knowing it will get replicated to the other side correctly. I’ve heard that MM pairs don’t make sense since you’re executing everything twice. It’s true that you are executing everything twice, but you’re doing it already if you’re using a master-slave(s) setup, and the benefits that come from MM pairs are huge. In addition to giving you fault tolerance and load balancing, they are the key to being able to do non-disruptive, live schema changes.

The other part of the puzzle is our shard-aware software layer: our in-house built ORM. It does many different things, but for our current topic, it finds where in our shards a particular object’s data lives. Whenever we need to access the data for an object, the ORM first goes to one of two “index” servers we have, then go to the shard that has the needed data. These index servers are also a MM pair. Index servers get a very large amount of queries, but they are all extremely fast, all in the order of 10-100𝜇s. It’s common for sharded architectures not to have an index server. You simply decide on a sharding scheme when you start, say by user id, then divide the data among your shards knowing where ranges of users live. Everything works great until the number of users on a shard grows beyond what one shard can handle, and by then you’re already in trouble. By having an index server, we can move data between shards and simply update the index to point to the new location.

Our ORM reads a configuration file when it starts, that among other things, contains the list of shard servers available to it. We can add shards as needed with time and add them to the configuration file to start writing data to them, also migrating users so new shards are not idle at first and to balance the load among all shards.

The kicker: when we do schema changes, we take out one server from each of the MM pairs from the configuration file and gracefully restart the application. The ORM re-reads its configuration and knows only about the active shard sides. This leaves the application running on half of our database servers. Nobody notices. We immediately see in our many graphs that one side’s traffic plummets and the other side is taking all the load.

Note that half of the servers does not mean half of the data. All data lives on both sides of a MM pair. Replication is still going both ways, we never break it. The active side simply stops getting inserts/updates/deletes from the inactive side because nothing is connecting to it. But the inactive side still gets inserts/updates/deletes from the active side since it’s still a slave. We could break replication for the ALTERs, but there’s no benefit in doing so and adds an unnecessary step (with the one exception of the session we’re actively doing ALTERs in. We don’t want those to replicate.)
At this point we are ready to make as many changes as we need on the inactive side. In Mysql terms, ALTERs. These ALTERs can take anywhere from minutes to hours to complete and lock the tables they are modifying, but we’re operating on the inactive side and definitely don’t want any of our work to replicate to the other side, so we prepend ALTERs with SET SQL_LOG_BIN=0.
When these alters are done, they have been applied to the inactive side only.
Another change in the config file places these servers back into active mode. We wait for load to stabilize between both sides, replication to catch up if it has lagged behind, then we’re ready to repeat for the side that hasn’t been ALTER’ed.

Taking sides out of production is not only useful for schema changes, but for upgrades, configuration changes, and any other necessary downtime.

So this is all great, works well for us. We routinely do schema changes with no user impact. But what if you don’t have an ORM? Mysql Proxy may be your answer. It’s very simple to have web servers connect to a pool of available backend database servers with Mysql Proxy. You can read the documentation for it at Mysql’s website. An important feature of Mysql Proxy is that it allows you to change configuration on-the-fly, so you can take servers in and out without even having to stop or reload your application.

MM pairs have had a bad reputation of being quirky. They can be, but as long as you don’t let your database generate anything, they work. When you need to do frequent schema changes in a 24×7 environment, they are key to no-downtime schema changes.

If you want more details on our database architecture, you can also check here

“Today, in conjunction with Hacker School, Etsy is announcing a new scholarship and sponsorship program for women in technology: we’ll be hosting the summer 2012 session of Hacker School in the Etsy headquarters, and we’re providing ten Etsy Hacker Grants of $5,000 each — a total of $50,000 — to women who want to join but need financial support to do so. Our goal is to bring 20 women to New York to participate, and we hope this will be the first of many steps to encourage more women into engineering at Etsy and across the industry.” – find out more, and then checkout Etsy Hacker Grants

The first set of servers we installed had been running happily, however this new set were not. They would always kernel panic under the slightest load, and sometimes without much load at all.

A nice feature, which is enabled by default in CentOS 6, allows your kernel to dump a core file when it panics. When your system comes back, you can retrieve this core file, examine it and work out what happened. This is a short story on how we did it. The work we did is on CentOS but it can easily be applied to other Linux distributions too.

Basics and theory

When a kernel panics, it dumps a core file into /var/crash/, which can then be examined.
In your tool belt, you need:

• /usr/bin/crash (installed through the “crash” package)
• A debuginfo kernel (downloaded and installed from http://debuginfo.centos.org/6/x86_64/)
• The vmcore file from /var/crash/....

crash is gdb-like. It uses gdb and lets you examine vmcore files from kernels.

Basic debugging

Start crash by pointing it at the vmlinux file installed by the debuginfo kernel, and the vmcore file:

sudo crash /usr/lib/debug/lib/modules/`uname -r`/vmlinux \
/var/crash/<time>/vmcore

You will see output like this:

      KERNEL: /usr/lib/debug/lib/modules/<kernel>/vmlinux
    DUMPFILE: /var/crash/127.0.0.1-2012-03-28-23:51:01/vmcore
  [PARTIAL DUMP]
        CPUS: 16
        DATE: Wed Mar 28 23:50:56 2012
      UPTIME: 00:23:26
LOAD AVERAGE: 0.95, 1.45, 1.01
       TASKS: 986
    NODENAME: buildtest07
     RELEASE: 2.6.32-220.7.1.el6.x86_64
     VERSION: #1 SMP Wed Mar 7 00:52:02 GMT 2012
     MACHINE: x86_64  (2400 Mhz)
      MEMORY: 24 GB
       PANIC: "Oops: 0000 [#1] SMP " (check log for details)
         PID: 0
     COMMAND: "swapper"
        TASK: ffff880337eb9580  (1 of 16)
[THREAD_INFO: ffff880637d18000]
         CPU: 12
       STATE: TASK_RUNNING (PANIC)

This tells us some important bits of information:

• The command being run was swapper. swapper is a kernel process responsible for scheduling time on the CPU. When a panic happens here, there’s a likelihood that we’re looking at something in the kernel space that broke, rather than something in user space.
• The panic was an Oops.
• There are dates, times, number of running processes and other handy information too.

If you now run:

crash> log

and jump to the end, you will see much longer output like this:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
IP: [<ffffffff8142bb40>] __netif_receive_skb+0x60/0x6e0
PGD 10e0fe067 PUD 10e0b0067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/block/dm-6/removable
CPU 12
Modules linked in: veth bridge stp llc e1000e serio_raw
                   i2c_i801 i2c_core sg iTCO_wdt
                   iTCO_vendor_support ioatdma dca i7core_edac
                   edac_core shpchp ext3 jbd mbcache sd_mod
                   crc_t10dif ahci dm_mirror dm_region_hash
                   dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 0, comm: swapper Not tainted <kernel> #1 Supermicro X8DTT-H/X8DTT-H
RIP: 0010:[<ffffffff8142bb40>]  [<ffffffff8142bb40>] __netif_receive_skb+0x60/0x6e0
RSP: 0018:ffff88034ac83dc0  EFLAGS: 00010246
RAX: 0000000000000060 RBX: ffff8805353896c0 RCX: 0000000000000000
RDX: ffff88053e8c3380 RSI: 0000000000000286 RDI: ffff8805353896c0
RBP: ffff88034ac83e10 R08: 00000000000000c3 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000015 R14: ffff88034ac93770 R15: ffff88034ac93784
FS:  0000000000000000(0000) GS:ffff88034ac80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000060 CR3: 000000010e130000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff880637d18000,
task ffff880337eb9580)
Stack:
 ffffc90013e37000 ffff880334bdc868 ffff88034ac83df0 0000000000000000
<0> ffff880334bdc868 ffff88034ac93788 ffff88034ac93700 0000000000000015
<0> ffff88034ac93770 ffff88034ac93784 ffff88034ac83e60 ffffffff8142c25a
Call Trace:
 <IRQ>
 [<ffffffff8142c25a>] process_backlog+0x9a/0x100
 [<ffffffff814308d3>] net_rx_action+0x103/0x2f0
 [<ffffffff81072001>] __do_softirq+0xc1/0x1d0
 [<ffffffff810d94a0>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff8100c24c>] call_softirq+0x1c/0x30
 [<ffffffff8100de85>] do_softirq+0x65/0xa0
 [<ffffffff81071de5>] irq_exit+0x85/0x90
 [<ffffffff814f4dc5>] do_IRQ+0x75/0xf0
 [<ffffffff8100ba53>] ret_from_intr+0x0/0x11
 <EOI>
 [<ffffffff812c4b0e>] ? intel_idle+0xde/0x170
 [<ffffffff812c4af1>] ? intel_idle+0xc1/0x170
 [<ffffffff813fa027>] cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009e06>] cpu_idle+0xb6/0x110
 [<ffffffff814e5ffc>] start_secondary+0x202/0x245
Code: 00 44 8b 1d cb be 79 00 45 85 db 0f 85 61 06 00 00 f6 83 b9
      00 00 00 10 0f 85 5d 04 00 00 4c 8b 63 20 4c 89 65 c8 49 8d
      44 24 60 <49> 39 44 24 60 74 44 4d 8b ac 24 00 04 00 00 4d
      85 ed 74 37 49
RIP  [<ffffffff8142bb40>] __netif_receive_skb+0x60/0x6e0
 RSP <ffff88034ac83dc0>
CR2: 0000000000000060

At this level the most interesting thing to note is the line starting with BUG:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000058

This tells us why the panic happened: A NULL pointer dereference.

Intermediate debugging

Now some more gnarly stuff. If you’re still here, you’re pretty damn brave.
There some cool things we can do in crash which might help us.
For example, we can look at the list of running processes:

   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
      0      0   0  ffffffff81a8d020  RU   0.0       0      0  [swapper]
>     0      0   1  ffff880638628a80  RU   0.0       0      0  [swapper]
>     0      0   2  ffff880337d934c0  RU   0.0       0      0  [swapper]
>     0      0   3  ffff8806386294c0  RU   0.0       0      0  [swapper]
>     0      0   4  ffff880337dd3580  RU   0.0       0      0  [swapper]
>     0      0   5  ffff880637c84080  RU   0.0       0      0  [swapper]
>     0      0   6  ffff880337df1540  RU   0.0       0      0  [swapper]
>     0      0   7  ffff880637c84ac0  RU   0.0       0      0  [swapper]
>     0      0   8  ffff880337e33500  RU   0.0       0      0  [swapper]
      0      0   9  ffff880637c85500  RU   0.0       0      0  [swapper]
>     0      0  10  ffff880337e774c0  RU   0.0       0      0  [swapper]
>     0      0  11  ffff880637cf40c0  RU   0.0       0      0  [swapper]
>     0      0  12  ffff880337eb9580  RU   0.0       0      0  [swapper]
>     0      0  13  ffff880637cf4b00  RU   0.0       0      0  [swapper]
>     0      0  14  ffff880337ed7540  RU   0.0       0      0  [swapper]
>     0      0  15  ffff880637cf5540  RU   0.0       0      0  [swapper]
      1      0   8  ffff880638628040  IN   0.0   21364   1568  init
      2      0   8  ffff880337c714c0  IN   0.0       0      0  [kthreadd]
      3      2   0  ffff880337c70a80  IN   0.0       0      0  [migration/0]
      4      2   0  ffff880337c70040  IN   0.0       0      0  [ksoftirqd/0]
      5      2   0  ffff880337c99500  IN   0.0       0      0  [migration/0]
....

The list continues on for another 950 lines. The lines starting with a > indicate the processes currently active on a CPU. We are working on a 16 core system, and 14 of them are currently in kernel mode tending to CPU scheduling duties. (The other 2 further down were gmond and php.)

You can also change the context you’re currently in. For example, currently we’re looking at the CPU where the kernel panic happened. However, we can switch to another CPU if we wanted to using set -c <CPU> and examine things there. (We don’t want to do this yet.)

Speaking of our kernel panic! Let’s get back to that..
The code at the end of the previous section tells us a null pointer dereference was the cause of the crash, and it gives us some clues as to what happened:

IP: [<ffffffff8142bb40>] __netif_receive_skb+0x60/0x6e0

Here’s the code that was being executed! __netif_receive_skb, this sounds like it might have something to do with receiving things on the network interface, if I had to guess! (and if I did, I’d be right – many things in the kernel are named to be obvious).

We could even grep through the kernel source to find this!

shell> ack -a __netif_receive_skb
net/core/dev.c
    2705:int __netif_receive_skb(struct sk_buff *skb)

Looking at line 2705 of net/core/dev.c shows us the method that was running when things broke down.

Advanced debugging

Are you really still reading this? You’ve just gone from brave, to foolish! There is no turning back! Here’s the rabbit hole:

There’s a lot more information here! Just because we know the method that was invoked doesn’t help us that much. The method could be 2 lines and the solution could be clear, but more likely the method will be 100 lines long and make many references to other things. So let’s start poking at the memory to see what specifically happened:

IP: [<ffffffff8142bb40>] __netif_receive_skb+0x60/0x6e0

There are two each bits of information in this line:

1. ffffffff8142bb40 is the memory address at which the problem occurred.
2. 0x60 is the hex offset for the line in the memory that caused the problem. 0x60 is 96 in decimal – remember this!
   We can inspect the memory by doing this:

   crash> dis -rl ffffffff8142bb40
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../net/core/dev.c: 2706
   0xffffffff8142bae0 <__netif_receive_skb>:       push   %rbp
   0xffffffff8142bae1 <__netif_receive_skb+1>:     mov    %rsp,%rbp
   0xffffffff8142bae4 <__netif_receive_skb+4>:     push   %r15
   0xffffffff8142bae6 <__netif_receive_skb+6>:     push   %r14
   0xffffffff8142bae8 <__netif_receive_skb+8>:     push   %r13
   0xffffffff8142baea <__netif_receive_skb+10>:    push   %r12
   0xffffffff8142baec <__netif_receive_skb+12>:    push   %rbx
   0xffffffff8142baed <__netif_receive_skb+13>:    sub    $0x28,%rsp
   0xffffffff8142baf1 <__netif_receive_skb+17>:    nopl   0x0(%rax,%rax,1)
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../net/core/dev.c: 2715
   0xffffffff8142baf6 <__netif_receive_skb+22>:    cmpq   $0x0,0x18(%rdi)
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../net/core/dev.c: 2706
   0xffffffff8142bafb <__netif_receive_skb+27>:    mov    %rdi,%rbx
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../net/core/dev.c: 2715
   0xffffffff8142bafe <__netif_receive_skb+30>:    jne    0xffffffff8142bb16 <__netif_receive_skb+54>
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../arch/x86/include/asm/atomic_64.h: 23
   0xffffffff8142bb00 <__netif_receive_skb+32>:    mov    0xbdcbae(%rip),%eax        # 0xffffffff820086b4
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../net/core/dev.c: 1364
   0xffffffff8142bb06 <__netif_receive_skb+38>:    test   %eax,%eax
   0xffffffff8142bb08 <__netif_receive_skb+40>:    jne    0xffffffff8142c008 <__netif_receive_skb+1320>
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../net/core/dev.c: 1367
   0xffffffff8142bb0e <__netif_receive_skb+46>:    movq   $0x0,0x18(%rdi)
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../include/trace/events/net.h: 68
   0xffffffff8142bb16 <__netif_receive_skb+54>:    mov    0x79becb(%rip),%r11d        # 0xffffffff81bc79e8
   0xffffffff8142bb1d <__netif_receive_skb+61>:    test   %r11d,%r11d
   0xffffffff8142bb20 <__netif_receive_skb+64>:    jne    0xffffffff8142c187 <__netif_receive_skb+1703>
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../net/core/dev.c: 2719
   0xffffffff8142bb26 <__netif_receive_skb+70>:    testb  $0x10,0xb9(%rbx)
   0xffffffff8142bb2d <__netif_receive_skb+77>:    jne    0xffffffff8142bf90 <__netif_receive_skb+1200>
   /usr/src/debug/kernel-2.6.32-220.7.1.el6/.../include/linux/netpoll.h: 86
   0xffffffff8142bb33 <__netif_receive_skb+83>:    mov    0x20(%rbx),%r12
   0xffffffff8142bb37 <__netif_receive_skb+87>:    mov    %r12,-0x38(%rbp)
   0xffffffff8142bb3b <__netif_receive_skb+91>:    lea    0x60(%r12),%rax
   0xffffffff8142bb40 <__netif_receive_skb+96>:    cmp    %rax,0x60(%r12)

WOW! Lots of output!

What you’re looking at is the full list of steps that took place, starting with the method that was invoked and ending with the fault.
If you look at the last line of the output, you should recognise two key things:

0xffffffff8142bb40 <__netif_receive_skb+96>:    cmp    %rax,0x60(%r12)

1. The memory address is the one we peeked
2. The number 96 is the decimal offset in the memory where the fault occurred.
   A few lines above this, we see a new filename and line number mentioned, include/linux/netpoll.h:86.
   Congratulations! This is the actual line which caused the NULL pointer dereference!

But what is it actually DOING?

cmp    %rax,0x60(%r12)

Let’s examine this bit. In assembly, cmp is called to compare two registers. %rax and %r12 are two CPU registers which we want to compare. If we got back to our log output, we see the registers:

RAX: 0000000000000060 RBX: ffff8805353896c0 RCX: 0000000000000000
RDX: ffff88053e8c3380 RSI: 0000000000000286 RDI: ffff8805353896c0
RBP: ffff88034ac83e10 R08: 00000000000000c3 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000

RAX has some data in it, but R12 is NULL!

Now take a look at code that was mentioned earlier:

 84 static inline int netpoll_receive_skb(struct sk_buff *skb)
 85 {
 86         if (!list_empty(&skb->dev->napi_list))
 87                 return netpoll_rx(skb);
 88         return 0;
 89 }

If you know linux networking internals, a few things can be understood from this:

1. We’re dealing with New API (NAPI) code, based on the variable name napi_list
2. NAPI is invoked when the system thinks there are a large number of interrupts being handled, and changing to RX POLLING would be more efficient
3. sk_buff is the struct that holds data about linux sockets
4. We’re checking if the list, &skb->dev->napi_list is empty, so we’re expecting it to be defined, and be of the right type.

What is probably going on here, is that expecting &skb->dev->napi_list to hold a list of incoming packets that need to be processed. Unfortunately, it turns out that something, most likely napi_list, is not set, causing the NULL pointed dereference.

Congratulations! You’ve found the problem. Finding what the cause of this is, however, will be done in a future tutorial.
In the meantime we have some options for workarounds:

1. Try and test if napi_list is NULL just before the !list_empty check, and define it. This might have other unforeseen issues though – what if it causes a memory leak? It’s certainly not fixing the cause of the problem.
2. Disable the use of NAPI by recompiling the network driver. This is a pretty simple operation and probably worth trying, especially if the system isn’t massively CPU bound and dropping packets.

We ended up going with option 2, as it was the fastest and most likely fix.

But we’ll still be sleeping with one eye open…

Some more details here:
http://basho.com/blog/technical/2012/03/16/Drinkup-with-Etsy-on-March-20/

A new engineer’s first task is to snap a photo using our in house photo booth (handmade of course) and upload it to the about page. Everyone gets a shiny new virtual machine with a working development version of the site, along with their LDAP credentials, github write access, and laptop. We use an internal cloud system for the VM’s, mostly because it was the most fun thing to build, but also gives us the advantage of our fast internal network and dedicated hardware. The goal is a consistent environment that mirrors production as closely as possible. So what is the simplest way to build something like this in house?

We went with a KVM/QEMU based solution which allows for native virtualization. As an example of how you may go about building an internal cloud, here’s a little bit about our hardware setup. The hypervisor runs on HP DL380 G7 servers that provide us with a total 72G RAM and 24 cores per machine. We provision 11 guests per server, which allows each VM 2 CPU cores, 5G RAM, and a 40G hard drive. Libvirt supports live migrations across non-shared storage (in QEMU 0.12.2+) with zero downtime which makes it easy to allocate and balance VM’s across hosts if adjustments need to be made throughout the pool.

We create CentOS based VM’s from a disk template that is maintained via Openstack Glance, which is a tool that provides services for discovering, registering, and retrieving virtual images. The most recent version of the disk images are kept in sync via glance, and exist locally on each server for use in the creation of a new VM. This is faster than trying to pull the image over the network on creation or building it from scratch using Kickstart like we do in production. The image itself may have been kickstarted to match our production baseline, and we template a few key files such as the network and hosts information which is substituted on creation, but in the end the template is just a disk image file that we copy and reuse.

The VM creation process involves pushing a button on an internal web page that executes a series of steps. Similar to our one button deployment system, this allows us to iterate on the underlying system without disruption to the overall process. The web form only requires a username which must be valid in LDAP so that the user can later login. From there the process is logged such that it that provides realtime feedback to the browser via websockets. The first thing that happens is we find a valid IP in the subnet range, and we use nsupdate to add the DNS information about the VM. We then make a copy of the disk template which serves as the new VM image and use virt-install to provision the new machine. Knife bootstrap is then kicked off which does the rest of the VM initialization using chef. Chef is responsible for getting the machine in a working state, configuring it so that it is running the same version of libraries and services as the other VM’s, and getting a checkout of the running website.

Chef is a really important part of managing all of the systems at Etsy, and we use chef environments to maintain similar cookbooks between development and production. It is extremely important that development does not drift from production in its configuration. It also makes it much easier to roll out new module dependencies or software version updates. The environment automatically stays in sync with the code and is a prime way to avoid strange bugs when moving changes from development to production. It allows for a good balance between us keeping things centralized, controlled, and in a known-state in addition to giving the developers flexibility over what they need to do.

At this point the virtual machine is functional, and the website on it can be loaded using the DNS hostname we just created. Our various tools can immediately be run from the new VM, such as the try server, which is a cluster of around 60 LXC based instances that spawn tests in parallel on your upcoming patch. Given this ability to modify and test the code easily, the only thing left is to overcome any fear of deployment by hopping in line and releasing those changes to the world. Engineers can be productive from day one due to our ability to quickly create a consistent environment to write code in.

Dividing tests by common points of error made triaging failures systemic failures quick, and enticed everyone to write faster, more deterministic tests, but not all was perfect. Our Jenkins dashboard was quite verbose.

The numerous jobs on our dashboard were great for pinpointing where the failures were, but it was difficult to determine at which stage of the deploy pipeline the failures existed. Some tests were executed on every commit. Some tests were executed when the QA button was pushed. Some tests were executed against a freshly pushed Princess or Production build. We were using the Jenkins IRC plugin, and the number of messages per hour was drowning out necessary communication in the #push channel.

We needed some way to communicate the test status at each stage of the deployment pipeline.

We considered using Downstream Jobs, but fingerprinting was awkward and difficult to set up, and all-in-all it wasn’t quite what we were looking for.

We also considered Matrix Jobs, but a Matrix Job is designed to execute several jobs with parameter(s) varied along configuration vector(s), i.e. build node, operating system, browser, arbitrary parameter, etc. This was not a fit for the purpose because our jobs had wildly different configurations that could not be coerced into mere parameter differences.

What we needed was a way to create a Jenkins job type that would execute a selection of arbitrary Jenkins jobs, wait for the jobs to finish, and report a single result while still making it possible to drill down to sub-jobs to determine the sources of failures.

So we wrote a Jenkins plugin to achieve this, the Jenkins Master Project Plugin.

Now our Jenkins dashboard represents the deployment pipeline:

When a stage turns red (or yellow), you can click through to that particular Master Build, see what tests failed and drill through the results (or even rebuild).

We also wrote a Triggering User Plugin for determining the user who triggered the build and a Deployinator Plugin to link key Deployinator information to particular Jenkins build.

The Triggering User and Master Project plugins are both integral to our latest version of Try.

We have also made our Nagios plugin for Jenkins readily available on the Etsy GitHub account. We used this for experimenting with alerting on the health of Jenkins.

All of these plugins are freely available on GitHub under the Etsy organization. Enjoy!

If you’re a female developer or someone who wants to get more female developers involved in your organization, Austin All-Girl Hack Night and Girl Develop It invite you to come have brunch with us, talk about dev, and hopefully make some new friends! We’ll be serving a full breakfast, including coffee and breakfast cocktails, thanks to our gracious sponsors: Etsy, Bocoup, spire.io, Headspring, and TEKsystems.

Also, the RSVP is a shell, and invitation is executable, so save yourself the pain of trying to cat it like me.

Badge optional, RSVP mandatory, Unix skills strongly suggested to get through the RSVP, a willingness to drink breakfast cocktails recommended.

The Security and Defensive Systems group here at Etsy have flipped this model around.  Rather than warn the user when a malicious link is followed, we block the link (or the whole page) from displaying in the first place.

There are a few ways to use the Google Safe Browsing service. For lower volume queries, there is a very simple REST API.  For high volume, high performance systems, the GSB V2 protocol is more appropriate as it mirrors the entire GSB database locally. It’s designed to scale to an extremely large number of clients while minimizing network traffic.  To do so, it uses a complicated protocol involving multiple blacklists and whitelists sent as a series of distributed binary diffs.

While many implementations of the GSB protocols are available, for a variety of reasons they were not appropriate for use in Etsy’s operational environment (e.g. use of autoincrement ids, designed to run under a web server, etc), and so we created our own.  We have open sourced our version and made it available in our gsb4ugc git repository. It’s in PHP, but it should be straightforward to port to other languages, as it’s really more of a toolkit than a standalone product.

To use, you’ll need to create and assemble resources to create your own API. First you need to set up some boilerplate for both the GSB updater and client:

// Set up a db connection.
$dbh = new PDO('mysql:host=127.0.0.1; dbname=gsb', 'user', ‘password’);
$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

// Create storage; works with mysql, sqlite.
// No auto-increment IDs, so it's safe with master-master replication.
// Etsy subclasses this and adds StatsD calls. http://etsy.me/dQwVXi
$storage = new GSB_StoreDB($dbh);

// Create network access. Pass in your GSB API key. Uses PHP curl.
$network = new GSB_Request($api);

// Logger. Subclass to use your logging infrastructure (or not).
$logger = new GSB_Logger(5);

Then one needs to setup a cron job that runs every 30 minutes to start mirroring the GSB database.

$updater = new GSB_Updater($storage, $network, $logger);
$updater->downloadData($gsblists, FALSE);

It takes about 24 hours to full sync up. Finally, you are able to start checking URLs:

$client = new GSB_Client($storage, $network, $logger);
$url = "http://malware.testing.google.test/testing/malware/”;
print_r($client->doLookup($url));

should return something similar to:

[list_id] => 1
[add_chunk_num] => 70219
[host_key] => b2ae8c6f
[prefix] => 51864045
[match] => malware.testing.google.test/testing/malware/
[hash] => 518640453f8b2a5f0d43bc2251....
[host] => testing.google.test/
[url] => http://malware.testing.google.test/testing/malware/
[listname] => goog-malware-shavar

More details are in the bin/samples directory of our repository.

We are currently scanning a few types of user generated content in production. This is done asynchronously from the website so we don’t block the user experience, however we still care about performance. Almost all performance metrics here at Etsy measure maximum and minimum times, as well as 90th percentile and mean, and this is no exception. The peak times occur when a network call is required, otherwise, it’s typically 5ms.

Since this is security-related code, another goal of gsb4ucg is testability.  The protocol-parsing code is separated out from database and networking code, so it’s very easy to write unit tests. This also helps to explain how the code works. As you see below, we have some more work to do:

In addition to expanding test coverage and improving performance, we’d like to add MAC support, and to use it for more content types on Etsy.  We’d also like to add the results from PhishTank for completeness and redundancy.  Comments, bug reports, patches and pull requests are all welcome, but if this type of work interests you, consider doing it full time.

Now, go forth and browse and consume content safely!

The most important component of The Etsy Way is culture and that is as difficult to teach as it is important. To get a sense of how we think about culture, take a look at Optimizing for Developer Happiness, which includes a 24-minute video of a talk I did and a link to the accompanying slides.

Here are a few more links about culture:

• Scaling startups
• How does Etsy manage development and operations?
• Code as Craft: Building a Strong Engineering Culture at Etsy (slides)

With the culture bits explained, below are a few other key posts in the Etsy canon. All of these are inter-related with the culture, of course, and help reinforce it (remember it’s all about culture. Did someone say “culture”?):

Quantum of Deployment (Erik Kastner). We deployed code to production more than 10,000 times in 2011. If you wonder “how did they do that?” this post will tell you all you need to know.

Track every release (Mike Brittain). Here, we write about the methods we use to track the success of every code deploy with application metrics. This is part of the not-so-secret sauce.

Measure Anything, Measure Everything (Ian Malpass). We introduce you to StatsD, the open source software we built at Etsy to enable obsessive tracking of application metrics and just about anything else in your environment. The best part is you can download StatsD yourself and try it out.

Divide and Concur (Noah Sussman and Laura Beth Denker). By reading this post, you’ll learn about all the inner workings of our automated testing setup: what software we use (with plenty of links), how we set it up, and the philosophy behind it all.

We also have tons of slides from talks we have done, all available in the Code as Craft group on Slideshare.

And last but not least, we have an Etsy Github repository with lots of goodies.

Pretty much everything we write about above is open source (even the culture) so the motivated reader will find links to tips and actual software along the way to actually set things up on his/her own. If there’s anything you’d like to know more about The Etsy Way, just let us know in the comments. We’ll add it if we have it, and probably write it if we don’t.

As you can tell, a really important part of The Etsy Way is encouraging people on the team to contribute to open source, write informative and entertaining blog posts, and put together killer presentations. If you want to join the fun, we’re always hiring.

Michelle D’Netto and Lindsey Baron, February 23, Selenium 101 Workshop. Brooklyn, NY.

Laura Beth Denker, February 24, Scaling Communication via Continuous Deployment. London.

We’re sponsoring Devopsdays Austin, April 2nd and 3rd. Austin, TX. Look for us.

John Goulah, April 11th, Starts with S and Ends With Hard: The Etsy Shard Architecture. Santa Clara, CA

Michelle D’Netto, Stephen Hardisty and Noah Sussman, April 16-18, Handmade Etsy Tests and Selenium In the Enterprise: What Went Right, What Went Wrong (So Far). London.

Laura Beth Denker, May 22nd, Developer Testing 201: When to Mock and When to Integrate and It’s More Than Just Style. Chicago, IL