Proxying NGINX from a subdirectory through to a service

I was working on some ElasticSearch stuff, and came across this issue that I felt I should write down somewhere for the next time I’m working on this, because it took me a while to figure out.

If proxy_pass is specified with URI, when passing a request to the server, part of a normalized request URI matching the location is replaced by a URI specified in the directive:

location /name/ {

The important part is that last slash in the proxy_pass directive. If you suffix a proxy_pass value with a slash, it will replace anything in the location path with the path used in the proxy_pass directive, invisibly. Exactly what I needed!


Tagged | Leave a comment

Modifying the default retention for Graphite

This guy’s blog post saved me hours. I didn’t realize that when you change your storage aggregation for Carbon, you have to tell whisper to resize your existing metrics in order for them to pick up the new settings! I’m sure this is in the docs somewhere, but I totally didn’t catch it. Awesome post.

Any existing metrics created will not automatically adopt the new schema. You must use to modify the metrics to the new schema. The other option is to delete existing whisper files (/opt/graphite/storage/whisper) and restart for the files to get recreated again.

On top of this post, I went out and found a great one-liner for having whisper modify your existing metrics, here on Github


Tagged | Leave a comment

CentOS 7 & Varnish failing to start

Vanilla varnish fails to start after installation on CentOS 7 because of SELinux. I was getting this error when trying to start:

Failed to set permissions on ./ Operation not permitted
VCL compilation failed

Then it would fail. This is not obvious, but it’s due to SELinux. I found a bug report on RHEL over here.

To fix this, disable SELinux with this command

sudo setenforce 0

Then you can start varnish.

Posted in Automation | Leave a comment

Lead DevOps Engineer

Since May 2013, I’ve been with Thrillist Media Group, as a Web Applications Developer. Awesome place, awesome job, awesome people, all around the most excellent place I’ve ever worked.

A year after joining, I took on a new challenge there. I had been learning and making strides with automating our development environments. When I presented it to people, they loved it. So, they decided that I would be placed on the Developer Operations team, with the main idea being I would be the guy to automate not only the developer environment creation for all our services across tech, but also to automate the creation of staging and production environments for all the same as well.

I dove in, and I’ve loved every minute of it.

Fast forward a few months after transferring to DevOps, my supervisor took a new position elsewhere, and I was promoted to the lead of the department. I’ve refocused the efforts of my small team (myself and one other employee) into 3 facets I feel are best able to help the company: Automation, Monitoring, and Continuous Integration.

We use Ansible for automation. Fantastic tool, and we have yet to hit a wall in terms of what it can do.

Sensu is running our monitoring. We’re also planning on putting a graphing framework called Grafana in place before Black Friday to help us understand trends in our infrastructure.

Jenkins is our weapon of choice for Continuous Integration. Not only does it manage deployments, but honestly we’re using it for a lot more.

It’s been a really amazing summer and fall, and it looks to be getting even more interesting as we go into the holiday season with, our retail arm, looking to kill it on Black Friday and Cyber Monday. However, with these tools working for us, I’m confident we’ll pretty much be able to sit back and watch the sales roll in.

I’ll also be refocusing the goal of this blog, because DevOps is the stuff I’m working on day to day. I’ve already written a couple of Ansible roles that I’ll be posting here. (and probably on the community-run Ansible Galaxy, as well) I’ve also got an idea for a Jenkins plugin that I might take on, but I’ve got to figure out how to write those first.

Thanks for reading!

Posted in Musings | Leave a comment

Homeless coders – Trees for Cars mobile app

Fantastic read, and I wish this guy all the best.

Grand was approached by a young programmer named Patrick McConlogue in mid-August with a choice: Take $100 or take an opportunity to learn how to code. Grand, who had been homeless since 2011 after he lost his job at MetLife and was priced out of his neighborhood when a high-rise went up on the next block, didn’t hesitate.

I tell my friends all the time, coding is a skill that you can take with you wherever you go, and make a difference in whichever endeavor you choose to be a part of. It should really be treated more like a trade than anything else, akin to being an electrician or a plumber, because at the end of the day, you have knowledge that anyone could learn, and if you haven’t learned something new every day, you’re doing something wrong.

If this guy has assimilated the skills it takes to create a mobile app in both iOS and Android, he can probably get himself a job paying a very decent amount in the next couple of weeks, especially with the exposure he’s going to get from the launch of this app.

Tagged | Leave a comment

BruceBlog: VIM Syntax Highlighting for Scala: A Bash One-Liner

Started learning Vim recently, and It’s been about 2 months since I’ve made the switch and I still learn new things about it every day. At Thrillist, we use several different languages, including Scala, which is a new one to me. Had to go out and find a syntax highlighter for it, and stumbled across this excellent one-liner. Sharing below!

Just for posterity, here is a Bash one-liner to download the plugins to handle syntax highlighting for Scala:

mkdir -p ~/.vim/{ftdetect,indent,syntax} && for d in ftdetect indent syntax ; do curl -o ~/.vim/$d/scala.vim$d/scala.vim; done
This is the kind of stuff that I hope to find more of and put up here.
Tagged | 1 Comment

Kerrick Long : FTP is so 90’s. Let’s deploy via Git instead!

This is an excellent post about how to manage your code in a more efficient way. I’ve replaced most of my deployment workflow with Git in one way or another, and I’ve totally seen the benefits. However, there has not been “one tutorial to rule them all” yet for setting up a version controlled development environment that pushes to production via git hooks. It’s something worth looking into and I think I’ll start writing something for it.

Tagged | Leave a comment

Internal Server Errors, and future infrastructure.

So on the second day of my decision to write every day, I’m blocked from writing by my server updating and erroring out my WordPress instance. Let this be a lesson to me: don’t put custom php.ini configurations in user’s directory, ever. I did it ages ago for myself, and of course totally forgot that it was even there, just hanging out while the system php.ini got updated.

Best practices for system administration I’m particularly bad at. I work with a couple of developer operations guys, who if they knew about this, would slap me across the face. I know enough to be dangerous, and this time it bit me in the ass.

I’ve worked at Thrillist for about 7 months now, and learned so damn much; it might be time to move beyond the HostGator VPS. There’s no reason that I couldn’t transfer everything over to a more dedicated platform, with more RAM and CPU power, and be able to script out a hosting signup platform. I’d love to automate a setup that integrated NGINX rather than Apache, for example, and I’ve always wanted to learn more about setting up email. I’m also pretty sure that I could automate setting up Google Apps for your domain, given someone signed up for the service and just wanted to point DNS stuff.


Posted in Musings | Leave a comment

Writing more often to exercise the writing muscle

I’ve been lazy for a while, which happens a month or so after I “re-launch” my blog. Every time. To be honest I’m not sure what the issue is with myself; I could lean towards the fact that “I’ve got a lot going on” but that doesn’t really fly because I’ve got extra time on the bus on the way back and forth from work. I don’t like the “I’m lazy” description, because all in all I think I do pretty well. I think that my issue is that I don’t treat writing like a job. Other things I think of as “my job” always get done, usually without a problem and on time. So, I read this article today, and in an attempt to change this behavior, I’m going to give this method a shot.

It was likely just over a year or two ago when I read a productivity tip Jerry Seinfeld gave about having a giant wall calendar and making an X across every day that he wrote jokes. His goal then became to not break the chain. If 750words shut down or turned off the “streak feature” it would be harder to stay motivated. People love seeing progress, in stories and in themselves.

Writing Rules

I’m going to put a calendar at my desk, and every day by 9am I’m going to publish a post. The post is going to be 300 words minimum, and will be well edited and fully realized. Each one will include an image. Because images are cool. Some might embed Youtube videos, although that’s a stretch because watching videos of people writing code is boring.

I think that writing every day, because it’s my job is a mindset that I have to assimilate, and once I do, I’ll be able to breeze through it every day. However, in the mean time, it might make for some shaky posts. Long term though I want to develop my “writing muscle” so that I can spin it up at any time.

Wish me luck!

Tagged | Leave a comment

Returning Hierarchical Data in a Single (Postgre) SQL Query

Andrew Bender brings us a great little tip about PostgreSQL.

When working with data in your backend code, you often need quick access to the relationships within your data model. Including nested JSON objects in your result set is a great way to accomplish this naturally. This approach works particularly well if you’re interfacing with the database using node.js or another platform with fast JSON decoding.

This almost makes me want to take a look at Postgre to see if it’s feasible for a project I’m working on, but I wonder about the performance implications.

Tagged | Leave a comment