I’d love to be able to use Lambda to perform AWS API functions, like scaling ASG’s or granting and revoking permissions to resources. A knock-knock model for access to an EC2, which adds and then removes X hours later a CIDR to a security group, so that you can gain access to the resource for a short time.

See: Tutorial

Crucial Before we make this thing permanent, be sure to include an authorizer, otherwise the API Gateway endpoint is security-through-obscurity only.

We rotate a one-week On-Call shift. The first few days, as the On-Call Guy, you’re either dumbstruck by how many nuisance pages you get all night long or you don’t get bothered and sleep like a baby. It often takes several days before you’re ready to be pissed off by the all night long nuisance pages, but by then, you’re so busy heads down working day-to-day changes thta you don’t prioritize fixing the all night long nuisance pages. So, the system has stayed broke for years.

I keep threatening to fix it, but I never do. But I recently found this blog post here, that maybe outlines how to make a reasonable AWS Health Alerting System, so I’m going to spend some time now figuring this out.

Design Ideas

I envision a Lambda which reads a json object from an S3 bucket or a record from Dynamodb which contains configuration data, specifically, perhaps lists of alerts to ignore, convert to daily emails, alert on immediately etc. The Lambda would be inbvoked by AWS Health and would itself send the message onto SNS etc. to deliver to the On-Call Guy. Thus we filter the messages AWS Health gives us into 4 quadrants.

We have to remember that everyone is somewhere on the spectrum from idiot to expert

Watch this video for a primer:

Tech layoff captured on film

Some takeaways from this experience:

• the people who are firing you don’t know you, don’t know how many peers you’ve befriended, wooed, done favors for, owe you favors. They don’t know how many customers you have helped, how many veeps you think love you. They just have your name on a piece of paper, along with a script intended to communicate to you that you don’t measure up.
• no matter what you say, if this happens to you, your time on the clock is near it’s end. No amount of argumentation can change that.
• your seniority may or may not have any relevance, but being there a short time won’t matter. You haven’t had enugh time to learn the job and show your unique skills? Too bad.
• you better be prepared. One week your working your JIRA tasks, the next week you hear that everybody on your team are getting HR calls and getting laid off.

I’ve never been laid off or fired. I’ve always lept from one tree to another like that old iPhone game Doodle Jump, stasis for years, then sudden action, then stasis again. I’ve always managed to gain at least 20% in increases in compensation as I have moved, which has led to my current position where, were I to be let go now, my family would suffer. I’ve forgotten how to live on one income. Its been a long time since I had to control my appetites to make ends meet. I’m being painfully honest here.

But that being said, I know I will survive. That is the First Pillar in Tech Layoffs Land. You must retain confidence in your core skills, because those core skills are marketable. If you’re a COBOL programmer, getthefuckouttahere, OR get into legacy mergers and conversions. COBOL was a hot commodity 24 years ago, and you might still be able to find an old VAX system needing maintenance. But chances are, you have some coding skills, or you know how to do database normalization and sql queries, or you know JavaScript. You probably know your way around one or two cloud consoles. Your skills don’t suck. Don’t freak out. rake new_post[] But having skills is not enough. Potential employers need to be made aware of those skills somehow. So, keep that Resume up to date. You need to review that document every quarter, making sure new things you learned in the past months are in the Skills section. You need to keep a list of skills and accomplishments ready for your yearly performance review anyway, so you might as well keep it in your Resume.

Second Pillar: you can’t afford to find work, if you’re broke. So, right now, go open a new savings account in some weird web bank you will promptly forget about, and setup direct deposit to put $100 a month in it. You’re gonna need liquid cash if the HR lady comes calling. DO NOT expect a beefy severance. On that fateful day, the HR lady may forget to mention the beefy severance, because all it amounts to is a weeks salary or less.

Third Pillar: be social. You’ll get that next job more easily if you make your self and your skills known to the most people. I always move from job to job based on recommendations from friends. Hiring on to a new industrial cold, with little knowledge of the company culture, is a recipe for a future layoff. Instead, hire on to a company staffed with friends who are asking you to apply. When your current company is laying off, chances are there’s another company hiring, and perhaps your friends are going there, and perhaps they will invite you in. So be nice to your coworkers and ensure that your skills are well known to them.

Final Pillar: have fun. One of these days, they’re going to carve a rectangle in the ground, and place the remains of you in it. So, try to find some joy in the right-now. Who knows? Maybe if you were more friendly and seemed to enjoy yourself more at your last job, they might have kept you around.

She is so great.

And I’m a dad. Being a dad means I already have daddy-dos and honey-dos numerous enough to occupy my time from here till my natural death all on their own. But as an honest, red-blooded American who grew up in the age of Woz and DHH and Spock and Dominic Giampaolo and Cyril Meurillon, I can’t just confine my aspirations to simple tasks. That’s why I have more prototype boards than I’ll ever solder, more wire than I’ll ever need for bodges, more technical manuals than I’ll ever read, not to mention kindle files outlining White Hat Hacking, how-to-build-a-robot guides, etc. etc. etc.

I’m into coding. I’m into speelunking and tunneling. I like woodworking and metal working and concrete working and house building. I like to read, and write, and compose music, and dungeon master. I’m addicted to Youtube and Podcasts and Kindle and Reddit. I have to set an alarm to go to bed, otherwise I’d find myself tinkering at 4:00am and getting no decent sleep for weeks at a time. Once I spent an entire month building sawhorses.

I have a 5HP 10” Grizzly table saw, I want a 12” one. I just bought a dual 48-core XEON workstation with 256Gb of RAM, based on the Skylake platform, I want one with a newer processor. I want a metal lathe and mill. I want a dump truck and an excavator and a hundred acres in the Rockies where I can dig tunnels and underground chambers. I drink two pots of coffee each morning, I want to learn how to make the coffee taste better than burned sadness.

Hoarding? I’m way past that! I have more beryllium (in the form of magnetrons) than the EPA would probably allow if they knew. I’m inspired, not repulsed, by The Radioactive Boyscout. I have a special way I shimmy through my piles and stacks so as to not fall into them and never be heard from again. And I dream of building the warehouse at the end of Raiders, where I can build shelves as far as the eye can see, and stash all my junk.

Anyway, I say all that to say this: fear not, if you’re just like me. I’ve figured out that life has meaning only if you lose yourself in your projects. One day, they’ll have to clean up all this mess. I’m hoping they find some value in my treasures.

And all the boxes marked DO NOT DROP - don’t drop them, seriously.

I discovered how complex the setup of a github pages website can be, but eventually I conquered it, which is encouraging, since I am 10 years older that I was when I setup the original site. 10 years, you say?

Oh yeah, so I was poking around on my old site, and I notice that the last Post’s creation date was Aug 22, 2013! This was startling, to say the least.

A lot has happened in the past 10 years. I changed jobs, got promoted to Principal, My kids graduated High School, or are in striking distance, anyway. My oldest daughter is a year away from graduating college!

We dealt with a Trump Administration, we had a Pandemic that killed millions and still lingers, infecting new folks and keeping others sick with Long Covid. We shifted pretty much all tech work to Work at Home, which is an extraordinary event in itself. Truly, it is hard to comprehend just how much has changed in the past 10 years.

Check out the github-pages-with-docker Youtube Video for a hella good video detailing how you should setup a github pages using jekyll. It’s an excellent walkthrough.

To add new posts, simply add a file in the _posts directory that follows the convention YYYY-MM-DD-name-of-post.ext and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.

Jekyll also offers powerful support for code snippets:

def print_hi(name)
  puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.

Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo. If you have questions, you can ask them on Jekyll Talk.

The product would integrate with puppet and activemq, and provide a secure, fault-tolerant self-service system for users (developers) who were not specifically trained to do puppet deployments to production servers. Since the developers are prone to make bonehead mistakes, and since the cloud environment was inherently a bit chaotic and non-deterministic (more on this later), the machine had to be able to filter out noise from the user end, and handle static on the server end with grace. It had to be designed from inception with these constraints in mind. It was a challenge and I’m happy with the results.

A Word on Puppet and ActiveMQ

Normally, people configure their instances to have puppet agents that kick off at regular intervals, enforcing configurations and policies like sentries marching in a circle around their post. The benefit of this kind of system is that it makes management feel warm and fuzzy - it’s a set it and forget it strategy. At any point in time, you can ask ‘what does this configuration look like on that bank of instances there?’ and the answer is always, ‘Lets look at the manifests.’ And if there is any question about a particular instance, that it might be somehow showing symptoms of differing from the policy, you can either wait for the next puppet refresh or trigger the refresh now. Further, puppet logs can give you a good view into the operation of the servers. You can add the puppet log to your syslog/graylog server and have a nice story to watch unfold in front of you.

However, if you don’t like the idea of a poke in the eye at regular intervals (especially in production), you might want some control on when the puppet agent runs. Having the puppet agent triggering every 15 minutes works if and only if your process is well-tested and well-known and if your puppet manifests are spot on. If any non-determinism lives comfortably in your process, you might want to consider turning that puppet agent daemon off.

Confession time: we had less testing that we were comfortable with, and we had more non-determinism that we would like to admit. And we had a huge, monolithic application to deploy with a lot of ghetto code. So we wrapped the puppet agent refresh in an activemq listener so we could trigger refreshes to one or many instances with pretty good control when we wanted. We then added in-service/out-of-service control to the scripts by integrating with the F5’s SOAP interface. We had the whole thing scripted and it worked great. But the only people who could work it were engineers who knew how it worked. In the end, for us, it was all work work work.

Enter: Rails and Sidekiq

I know how to build Rails apps. I can spin one up in 15 minutes. I can roll a rails app into an RPM and deploy it anywhere in the snap of a finger. I thought I’d try to build a rails interface into this puppet/activemq system, so we could allow developers and managers who didn’t have our puppet training to deploy software - even to production. It ought to ‘just work’, it ought to protect the company’s services and the client experience, but be able to take action when we wanted.

In order to model the deployment of a package to our environments, I had to map out the state machine we had created. I used the Workflow gem for this. In order to enable fault-tolerance, retry’s and run time customization, I added the Sidekiq gem and service. Sidekiq uses Redis on the backend, and allows you to run ruby scripts in a multi-threaded non-blocking fashion. It all runs pretty well.

Further Improvements

As we ran the system for a while (174 thousand deployments to date), we started running continuous integration with some applications. This involved triggering builds with Jenkins which generated an RPM, then detecting if new RPM’s were available, and if so, building a deployment for them and starting it automatically. We can now deploy from certification, through a variety of Unit tests, UI testing using selenium, sanity checks etc. all the way to production in 11 minutes. It beats all the work we had to accomplish once upon a time just to put a package in orbit.

Automation

I created a ‘build a VM’ script, containing the following:

# /setup/deploy_vm.bash

source /setup/lib.bash

if [ $# -eq 0 ]; then
    echo "No arguments supplied!"
    echo "REQUIRED: 1 = instid [example: centos-git], 2 = insthost [example: centos-git]"
    exit 1
fi

instid="$1"
insthost="$2"
tzone="US/Central"
domain="atomic.org"
discsize="20G"
memsize="2048"

normallog "grab centos 7 cloud image"

cd /var/lib/libvirt/boot

if [ -f "CentOS-7-x86_64-GenericCloud.qcow2" ]; then
    errorlog "qcow file exists, Skipping"
else
    detaillog "qcow file does not exist.  Download it"
    wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
fi

normallog "setup directories"

D=/var/lib/libvirt/images
VM=$instid
mkdir -vp $D/$VM

normallog "setup metadata"

cd $D/$VM

rm -f meta-data
touch meta-data

echo "instance-id: $instid" >> meta-data
echo "local-hostname: $insthost" >> meta-data

normallog "current meta-data:"

cat meta-data

normallog "setup user-data"

if [ -f ~/.ssh/id_$insthost ]; then
    errorlog "~/.ssh/id_$insthost exists.  Skipping"
else
    ssh-keygen -t ed25519 -C "VM Login ssh key foo" -f ~/.ssh/id_$insthost -P ""
fi

tmpkey=$(cat ~/.ssh/id_$insthost.pub)

cd $D/$VM

rm -f user-data
touch user-data

cat > user-data << ENDOFFILE
#cloud-config

# Hostname management
preserve_hostname: False
hostname: $insthost
fqdn: $insthost.$domain

# Users
users:
    - default
    - name: jason
      groups: ['wheel']
      shell: /bin/bash
      sudo: ALL=(ALL) NOPASSWD:ALL
      ssh-authorized-keys:
        - $tmpkey

# Configure where output will go
output:
  all: ">> /var/log/cloud-init.log"

# configure interaction with ssh server
ssh_genkeytypes: ['ed25519', 'rsa']

# Install my public ssh key to the first user-defined user configured
# in cloud.cfg in the template (which is centos for CentOS cloud images)
ssh_authorized_keys:
  - $tmpkey

# set timezone for VM
timezone: $tzone

# Remove cloud-init
runcmd:
  - systemctl stop network && systemctl start network
  - yum -y remove cloud-init
ENDOFFILE

cat user-data

normallog "Copy cloud image"
cd $D/$VM
cp /var/lib/libvirt/boot/CentOS-7-x86_64-GenericCloud.qcow2 $VM.qcow2

normallog "Create $discsize disc image"
cd $D/$VM
export LIBGUESTFS_BACKEND=direct
qemu-img create -f qcow2 -o preallocation=metadata $VM.new.image $discsize
virt-resize --quiet --expand /dev/sda1 $VM.qcow2 $VM.new.image
cd $D/$VM
mv $VM.new.image $VM.qcow2

normallog "Creating cloud-init iso"

mkisofs -o $VM-cidata.iso -V cidata -J -r user-data meta-data

normallog "Create a pool"

virsh pool-create-as --name $VM --type dir --target $D/$VM

normallog "Install VM"

cd $D/$VM
virt-install --import --name $VM \
--memory $memsize --vcpus 1 --cpu host \
--disk $VM.qcow2,format=qcow2,bus=virtio \
--disk $VM-cidata.iso,device=cdrom \
--network bridge=virbr0,model=virtio \
--os-type=linux \
--os-variant=centos7.0 \
--graphics spice \
--noautoconsole

normallog "Cleanup"

cd $D/$VM
virsh change-media $VM hda --eject --config
rm meta-data user-data centos7-vm1-cidata.iso

normallog "Get IP Address"

virsh net-dhcp-leases default

Test

I tested the script a few times: