How to use Virtual Machines for Privileged Access


Objective

Separate all privileged access work from non-privileged access work as a security measure. Directly accessing servers or administrative web pages from the same machine that you read email or do web browsing is a potential security risk. To mitigate this risk, create a “privileged access workstation”, or PAW, to support all work that needs to be conducted securely.

Constraints

Separate Privileged Access Workstation (PAW)

There needs to be separation between the machine hosting the PAW and the machine used for email and normal browsing. In other words, running a VM to act at the PAW on a computer that is used for email defeats the purpose. Therefore the PAW needs to be distant from the computer used for email, etc.

In our AWS Service Catalog we have a product that creates a Linux workstation that can be access via ssh or remote desktop, provided the connection is through the VPN. The VM is only addressable via the private network; it has no public IP address. This EC2 instance satisfies the separation requirement.

Virtual Private Network (VPN)

Access to any device on the internal network requires the use of a VPN. The VPN required is Global Protect from Palo Alto. It is configured as a full tunnel VPN, so all traffic from the device connected will flow through the VPN. In practice this has caused periodic issues with applications like Zoom, Office 365, and has resulted in some performance degradation.

It is possible to write a script that splits the tunnel, routing only work related traffic through the VPN and all other traffic through the gateway address the host machine has. This script is fragile, however, as Global Protect, openconnect, and even the local operating system, all can and will have updates that break the script. It is a “high cost” solution that is outside of any support provided by the security team or the local help desk. Using the native client for GP is the least objectionable option.

Hardware

While using two separate computers, one for secure activities and one for non-secure activities would be a potential solution, space constraints, not to mention maintaining two separate physical computers, makes this solution less desirable. Instead, use a single computer capable of running a virtual machine.

Solution

I’m opting to use two virtual machines, one locally hosted on my desktop and the other hosted at AWS. The local VM will have the Global Protect native client installed, and will be used solely to establish the VPN connection. ssh and RDP traffic will flow through this machine, to and from the AWS VM.

Hosting a local VM for the VPN isolates it from any local activities, such as Zoom and Office 365. This local VM isn’t resource intensive, as it won’t be doing any work; it’s a pass through to the AWS VM.

The remote VM, hosted at AWS, is where all the privileged work will occur. It has a private network address and therefore doesn’t require the VPN for any of the activities it will perform.

The final topography is a local desktop, running MacOS 12, that has an Ubuntu 20.04 VM running in VirtualBox. The Ubuntu VM has the Global Protect client installed. At the start of the day this client is used to establish a VPN connection, which has a 12 hour time limit. Through a ssh connection using port forwarding, both ssh and RDP traffic are directed through the local VM to the AWS VM.

Setup and Configuration

Local VM

I installed the latest version of VirtualBox and downloaded an Ubuntu 20.04 Desktop ISO. I setup the VM with 4 GB of RAM and 30 GB of storage. I created a user account with the same name as my work account, and I generated an ssh keypair.

ssh-keygen -t ed25519 -o -a 100 -f ~/.ssh/id_ed25519 -C "email@example.com"

Next I installed the Linux Global Protect client. With it installed I can establish a VPN session.

Remote VM

We created a Service Catalog entry that creates a Linux administrator workstation. It is hardened to some extent, does not allow ssh via password, and creates a key pair. This workstation can be configured with all of my tools and settings.

Local Host

From my desktop I can now ssh to the local Ubuntu VM, and then ssh to the AWS workstation. It is possible to chain the two ssh commands together into one command using the -t flag. If the local VM is called foo and the AWS VM is called bar the ssh command to sign in looks like this.

ssh -t foo \ ssh -t bar

Remote Desktop (RDP) works over port 3389. Adding some port forwarding to the ssh command it is possible to pipe the RDP traffic from the local host to the AWS VM. The new command now looks like this.

ssh -t -L 3389:localhost:3389 foo \ ssh -t -L 3389:localhost:3389 bar

This command can be shortened by using the AWS VM fully qualified domain name in place of localhost in the first half of the command.

ssh -t -L 3389:bar.aws.tld:3389 foo \ ssh -t bar

In my RDP client (Microsoft Remote Desktop) I created an entry for localhost. Once the ssh command has been issued, I can open an RDP session from my local machine that resolves on the AWS VM.

Daily Usage

My daily usage pattern follows these steps.

  1. Sign into the local Ubuntu VM
  2. Establish a Global Protect VPN session using the native client
  3. From a terminal on the local host run the ssh command to forward port :3389 to the AWS VM
  4. From a terminal on the local host establish any additional ssh session I desire, leaving off the port forwarding.

I’ve been using this setup for several days now, and it is working smoothly. By keeping the resolution on the RDP session relatively small, 1920x1080, there isn’t much lag while using it. I have a tmux session on the AWS VM that I connect to once I’m signed in, so that my session is always there.

Any privileged activity that requires a command line happens in a terminal on my local machine, that is connected via ssh through the local VM to the AWS VM. Any privileged web-based activity happens in a browser running on the AWS VM, accessed via RDP.

Summary

While it would be possible to eliminate the local VM, by running the VPN on the local desktop, separating these two concerns provides enough benefit to make the extra setup and configuration worthwhile. I have duplicated this setup on my work laptop, so I can connect in the same manner when I’m using it. Thanks to tmux I’m able to pick up the same session without any effort.

I may explore the command line Global Protect client to see if it is easier to use than the GUI one. I would still have to go to the local VM to establish the VPN session, but I could do it from the command line rather than through a GUI.


Writing an OS in Rust ⧁


Continuing the “write your own operating system” theme, here’s a series of articles talking about doing that with the Rush programming language.


Linux From Scratch ⧁


Something I want to do someday, because why not?


How to Write Idempotent Bash Scripts ⧁


Tips on writing a bash shell script that will not have side effects if run more than one time.


Cryptocurrency Deep Dive ⧁


A fascinating dive into the world of blockchain and cryprocurrency.


Graphite Grading Scale Explained ⧁


A nice explanation and visualization of the graphite scale used to categorize pencils.


Using Docker Compose to Run Jekyll


While setting up my new computer, one of the tasks I was faced with was recreating my Jekyll environment. Jekyll is the Ruby based static site generator I use for this site. MacOS has not always kept up with Ruby, and so I’ve had to employ rbenv to have an up to date Ruby installed. And to segregate the Gem dependencies Jekyll has from any other Ruby project that might have conflicting dependencies.

I kept putting off the task as I wasn’t looking forward to the possibility of some problems in Ruby Gem land. And then I thought about Docker—how hard would it be to use Docker as my Jekyll environment? As it turns out, not very hard at all. Eight lines of YAML total, not very hard.

A quick search lead me to this article, How to run Jekyll locally with Docker and Docker Compose. Using docker-compose and a Docker image that has Jekyll installed on it, you can quickly spin up a fully functioning Jekyll environment.

By putting the docker-compose.yml file in the root directory of my blog, and running

docker-compose up

I get Jekyll without having to install Jekyll, manage Gems, or use rbenv to control which version of Ruby and which collection of Gems is in use. Best of all, since Ruby is installed by default (version 2.6.3p62 as of this writing), I have rake at my disposal, so the Rakefile I have still works. This file has tasks for creating new postings, or new draft postings, for publishing drafts, and for deploying my site using rsync.

The only changes I made to the docker-compose.yml file described in the article, is that I used different flags on the jekyll serve command. I’m using

jekyll serve --watch --drafts

which is what I’ve had for a long time. The performance of the Docker image is a bit slower than it would be if it were natively installed. I may try --incremental at some point to see if that speeds things up.

Here’s my docker-compose.yml file.

services:
  jekyll:
    image: jekyll/jekyll:latest
    command: jekyll serve --watch --drafts
    ports:
      - 4000:4000
    volumes:
      - .:/srv/jekyll

Briefly, this file says, compose a Docker image using the latest jekyll base image. Once it’s ready run the Jekyll server, watching for new drafts, and expose the server on port 4000. Finally, map the current directory ., to /srv/jekyll in the image. Done.

Docker and docker-compose saved me a lot of time and effort. Now I wonder where else I could use it.


Solving Operation not Permitted Errors over Remote Access


Recently I purchased a 24" M1 iMac. One of my goals was to have a central place for all my digital content: movies, audiobooks, music, and photos. My venerable late-2013 MacBook Pro has a 1TB drive that cannot hold my entire collection along with my other files, applications, and the operating system.

For some time now I’ve been using an external USB drive to house my media. This drive has been plugged into an even more venerable 2009 MacBook Pro. The reason for using the much older laptop as host for my media is simple: that laptop doesn’t get moved. If the external drive were attached to my primary machine, I would constantly be moving it, with the laptop, or mounting and dismounting it.

With all my music, movies, and audiobooks offloaded to the external drive, there was enough room left on my primary laptop for all my photos, and the rest of my files. It wasn’t ideal, but it worked.

The new iMac has 256GB internal storage, and a pair of 4TB external drives. I have spent considerable time this past week consolidating all my digital media to one of the 4TB drives. I’ve also put my software projects directory, called code, there. When I am away from my desk and want to access something on the external drive on the iMac, those files are just an ssh connection away.

From the MacBook Pro I am using iTerm2 and ssh. From the iPadPro I’m using blink since it includes mosh, or mobile shell. The advantage to mosh is that the session stays in sync even if the connection is interrupted. Since iPadOS is fairly aggressive about stopping idle processes, using an ssh connection becomes tiresome as you are constantly having to reestablish it. With mosh, once you are connected you stay connected until you end the session.

I believe, though I am no longer certain, that the first remote session on my iMac was done from the iPadPro using mosh. On the iMac there is a component of mosh, mosh-server the runs and manages the incoming connection. This will play a key part in what happened with my ability to remotely access files over ssh/mosh.

MacOS allows you to control how your computer is shared. Under the Sharing preference pane in System Preferences, there is a Remote Login option. With that selected you can login to the local computer from a remote computer. Once Remote Login is enabled, there is a second option, Allow full disk access for remote users. This should be enough to permit ssh or mosh session access to any of the disks mounted.

In my case I was getting an Operation not permitted error any time I tried to access any external drive attached to the iMac. I got this error whether I used ssh or mosh, and from any remote device: iPadPro or MacBook Pro.

When the first remote login occurs a setting is added to Full Disk Access under the Security & Privacy Privacy pane. This setting is called sshd-keygen-wrapper. If it isn’t listed, or is listed and enabled, then full disk access is granted. If it is there and no enabled, however, full disk access is not granted. In my case, sshd-keygen-wrapper was present but not enabled.

I suspect that mosh (and mosh-server) having been the mechanism for the initial remote session, left the setting in a state where no remote access to external drives was allowed. Once I enabled that setting I was able to remotely access external drives using either ssh or mosh.

This Apple discussion thread, and the links it contains, do a better job of explaining the ins and outs of sshd-keygen-wrapper. https://discussions.apple.com/thread/250352651 I found this link through the ##apple IRC channel.


Kinds of Documentation


In the process of learning Go language programming I have consumed a lot of written and visual material. Today I listened to a few minutes of the Go Time podcast, where they were talking about documentation. The introduction to the episode really resonated with me, and so I wanted to capture it in my words here.

Tutorials

When first starting a new topic, particularly a technical one like, say Go programming, tutorials are a common starting point. Explanation mixed with samples, something you can follow along with at home. Learning my mimicking.

How To

How to resources are aimed at solving a particular problem. How to I set up an HTTP router in Go?, or How should a go project be structured? A how to is more narrowly focused. It aims to impart knowledge about this one specific topic.

Articles or Presentations

Somewhat broader in scope than a how to, an article or presentation (i.e., at a conference) dive a bit deeper into a topic. It isn’t a tutorial, assumptions are usually made about what you already know. And it isn’t a focused as a how to. It adds some though process to the solution or technique discussed.

Documentation

Documentation is a dictionary. An reference. You are looking up the specifics of an API or library. Depending on the maturity of the documentation, and whether it includes examples or not, finding information in the documentation can be difficult. If you know the package or library you need and only need to refresh your understanding of it’s parameters and return value, documentation is fantastic. If you are unsure, and are hunting for the right tool, documentation can be frustrating.

The Arc of Learning

In the course of learning a new programming language, I move through all of these categories of documentation. While working through a tutorial I may digress into a how to or look up things in the documentation. Generally though, I move from tutorials through how to and articles, and finally to documentation.


How Not to End a Customer Relationship


For more than eight years all of my sites, and my wife’s sites, were hosted at WebFaction. For $102 a year we had reliable, relatively straight forward to setup and use, web hosting and domain email. I originally signed up there as they would let you run Django or Ruby on Rails apps, and I was wanting to experiment.

About a year or so after signing up, I read an article online about tracking your own location using your phone, a database, and some mapping service I no longer remember the name of. I set it up, using PostgreSQL for the database (more experimenting), and once I had it working never really used it.

About two years ago WebFaction announced that their parent company had been bought by GoDaddy. I waited to see what kind of changes would result from this change of ownership. In October I learned that all the WebFaction sites would be migrated to tsoHost. Two days later I was told that might site would not be migrated, it would be terminated on December 9th. The reason given was PostgreSQL wasn’t supported on tsoHost.

A quick email to ask, “Hey, that PostgreSQL database you have that hasn’t seen any activity in years, can that go away? Otherwise we won’t be able to keep you as a customer.”, would have kept my business, at least for a while.

I spent a couple weeks in November preparing and then moving all our domains to Pair Networks. After backing up all the files on my WebFaction account, I deleted everything and let the account expire on December 9th.

The email telling me my account would be closed said that a refund for the unused portion of my annual fee would be refunded, and that this refund would take about 45 days to process. I created a calendar reminder and forgot about it.

When day 45 rolled around and I still hadn’t received any refund I contacted WebFaction. Their refund process requires PayPal. I don’t have, and don’t want, a PayPal account. Apparently they are unable to refund it to the credit card I used through PayPal’s payment portal. I asked if they could send me a check. Nope. Can’t do checks.

Today I got an email saying the could maybe do a wire transfer. They need the bank’s name, the account number, the BIC/Swift code, and the account name. Oh, and a screen shot of this information from the bank’s site. Any fees incurred for accepting the wire transfer, which I believe is coming from England, are on me.

I always liked WebFaction, but the way this account termination and refund have been handled have left me with a sour taste in my mouth. I’d advise against signing up with them, but they no longer exists, at least by that name.

Tomorrow I’ll gather all the numbers they need and send them off. I am not going to provide a screen shot of those details from the bank. And I’ll eat what ever fee my bank charges me for accepting an overseas wire transfer.

Online trolls exist because they can be anonymous. Companies, especially those that exist solely online, have some of the same characteristics. I can’t go to the WebFaction building and get my money, there isn’t a phone number, or a mail address. Just a URL that leads to a page with information posted pre-December 9th on it. Since there is little consequence, it is easy for a company to abuse their physical and metaphorical distance. All to their customer’s detriment.