How to Manage Bash Paths on MacOS


Viewing your bash path

The easiest way to view your bash path is to run

env

Then visually parse the output looking for the PATH entry and then read that line.

A slightly better way to view your bash path is to run

env | grep -i path

This will eliminate all the environment variables that aren’t paths.

$ env | grep -i path
SQLITE_EXEMPT_PATH_FROM_VNODE_GUARDS=/Users/mark/Library/WebKit/Databases
PATH=/Users/mark/.rbenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/MacGPG2/bin:/opt/X11/bin:/Users/mark/bin:/usr/local/sbin:/usr/local/go/bin:/Users/mark/.cargo/bin
GOPATH=/Users/mark/code/go

Personally I find parsing a list delimited by : to be difficult. So I use a simple function to break the path apart, displaying each entry on its own line, and color coding it along the way. Here’s the function:

path() {
local blue="\033[1;34m"
local green="\033[0;32m"
local cyan="\033[0;36m"
local purple="\033[0;35m"
local brown="\033[0;33m"
local reset_color="\033[0m"
  echo $PATH | tr ":" "\n" | \
    awk "{ sub(\"/usr\",   \"$green/usr$reset_color\"); \
           sub(\"/bin\",   \"$blue/bin$reset_color\"); \
           sub(\"/opt\",   \"$cyan/opt$reset_color\"); \
           sub(\"/sbin\",  \"$purple/sbin$reset_color\"); \
           sub(\"/local\", \"$brown/local$reset_color\"); \
           print }"
}

After setting some local variables to hold bash color codes, the function echoes the $PATH environment variable, using the tr, or translate command, to convert each : into a new line character, \n. The output of tr is passed to awk which colorizes the output but substituting some key words into the same keywords surrounded by color codes.

Here’s a sample output:

Image showing output of path function

One advantage to this output is it is far easier to find duplicates or path elements that are out of order.

A path_helper primer

macOS includes a utility called path_helper. The utility hopes to simplify management of your path. Knowing about it, and working with it, is the key to simplifying your path.

path_helper relies upon two sets of information. First is a list of standard paths found in /etc/paths. On my macOS computer /etc/paths contains:

$ cat /etc/paths
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

The second set of information used is the /etc/paths.d directory. Applications that make use of path_helper know to put a file at this location that contains their path element. Again from my computer, the contents of /etc/paths.d

$ ls -al /etc/paths.d
total 24
drwxr-xr-x    5 root  wheel   160B Sep 30  2018 ./
drwxr-xr-x  126 root  wheel   3.9K Sep 19 14:10 ../
-rw-r--r--    1 root  wheel    13B Oct 26  2016 40-XQuartz
-rw-r--r--    1 root  wheel    23B Feb  2  2015 MacGPG2
-rw-r--r--    1 root  wheel    17B Oct 25  2017 go

Looking at the go entry, for example, we see:

/usr/local/go/bin

Returning to the image showing the output of my path function, you can see the Go entry in the list. As well as entries for X-Quartz and MacGPG2. Also you can see the five standard path elements in positions 2 - 6. Obviously the /Users/mark/.rbenv/shims entry was pre-pended to $PATH. And it is also obvious that some things are getting added to the list twice.

Understanding that path_helper will seed $PATH with the standard elements, and any elements found in /etc/paths.d, it is now possible to clean up what is added to $PATH, and in what order they are added, in the bash configuration.

My bash configuration

My bash configuration is broken up into several files as the outgrowth of near constant tinkering and tweaking.

la -la ~/.dotfiles/bash/
total 136
drwxr-xr-x@ 12 mark  staff   384B Sep 19 14:12 ./
drwxr-xr-x@ 40 mark  staff   1.3K Aug 15 22:33 ../
-rw-rw-r--   1 mark  staff   7.3K Aug 25 22:11 bash_aliases
-rw-rw-r--@  1 mark  staff   389B May  5  2016 bash_bindkeys
-rw-rw-r--@  1 mark  staff   1.9K Jan 20  2016 bash_colors
-rw-rw-r--@  1 mark  staff   1.8K Mar 22  2016 bash_docker_functions
-rw-rw-r--   1 mark  staff    16K Aug 15 22:33 bash_functions
-rw-rw-r--   1 mark  staff   2.1K Sep 19 14:12 bash_profile
-rw-rw-r--   1 mark  staff   5.1K Sep 19 14:12 bashrc

(It is worth noting that the files are in my $HOME via symlinks.) The bash_profile file contains all the additions to my path.

I start off by clearing my $PATH and executing path_helper to seed to with the contents of /etc/paths and /etc/paths.d.

if [ -x /usr/libexec/path_helper ]; then
	PATH=""
	eval `/usr/libexec/path_helper -s`
fi

The -x conditional looks for an executable file; this allows the same bash_profile to be used on Linux machines without throwing errors.

For each element I want to add to $PATH I follow a similar pattern: test for the presence of the directory to be added, and if it is there, append it (or pre-pend it) to the variable. For example:

# set PATH so it includes user's private bin if it exists
if [ -d $HOME/bin ] ; then
  PATH="${PATH}:${HOME}/bin"
fi

Will add my personal bin directory to the list, at the end of the current list.

You can see my current bash_profile, and the rest of my bash configuration, in my GitHub dotfile repository. Here’s a sample from my bash_profile:

# set PATH so it includes user's private bin if it exists
if [ -d $HOME/bin ] ; then
  PATH="${PATH}:${HOME}/bin"
fi

# add Go path for Linux
if [ -d /usr/local/go/bin ] ; then
  PATH="${PATH}:/usr/local/go/bin"
  export GOPATH=$HOME/code/go
fi

# Enable path for Rust
if [ -d $HOME/.cargo/bin ] ; then
  PATH="${PATH}:$HOME/.cargo/bin"
fi

export PATH

Duplicate entries

Close reading of the path function output in the image above, and of the contents of /etc/paths.d, and of the sample bash_profile above, will show that the path for Go is in my $PATH twice. Once from the entry in /etc/paths.d and once from the lines in my bash_profile.

I have, and use, multiple computers running macOS. At work I have a pair of desktops and a laptop, and at home I have my personal laptop. Apparently I used a different method to install Go at work than I did at home. The method I used on my personal laptop created the /etc/paths.d entry, while the method I used at work did not. I can either remove the lines from bash_profile or add an entry to /etc/paths.d. Since I maintain my bash configuration I’ve decided to remove the entry /etc/paths.d (on any machine I find it) and rely upon the code in my bash_profile.

Conclusion

Here are my tips for keeping your path up to date and easy to work with.

  • Start with path_helper and add what you need in your $PATH to that initial set of elements.
  • Try to localize all the additions to your path in one place in your bash configuration.
  • Periodically display your path using the path function, and address anything the strikes you as out of place or no longer needed.
  • Pay attention to applications like Homebrew or rbenv that want to manipulate your path.

A Tale of Two Cards


Both my wife and I recently signed up for and got Apple Cards. The process was stunningly simple and quick. From our phones, and in about 2 minutes time, we were able to complete the application process, get approved, and have the new card in our respective digital wallets. In my case I had ordered an audio book a day or two earlier and within an hour or two of getting the Apple Card that charge happened on the new card generating 3% cash back.

The Apple Card application and approval process is fantastic. Using the card is also very nice. I’m rapidly learning which stores and restaurants accept Apple Pay. Starbucks can do Apple Pay at their drive through. Jimmy John’s can’t, so I park and go in side there. That way I get the card-less transaction 2%; at the drive through I’d only get 1% since I’d have to hand them my physical card.

Prior to getting these cards, both of us had a debit card with our bank, and we both have a VISA card through the bank too. We don’t use the VISA cards. For travel or large purchases we have a Citibank MasterCard. The Citi card is in my name. My wife’s Apple Card application resulted in a higher interest rate than I was offered. We both decided this was due to a lack of credit history under her name. Toward that end she applied for an Amazon VISA card in her name. This card offers 5% cash back on purchases made at Amazon or Whole Foods. We do a fair amount of shopping on an annual basis at Amazon, so 5% back on those purchases is a good deal.

The contrast in the process to get the card and have it fully setup is stark. While she was approved immediately, and was able add it to her Amazon account, she was not able setup an account with Chase until she had the physical card in her possession. While she can, and does, keep track of purchases, there was no way to see the account.

The new card arrived in the mail today - six days after she applied. This is actually pretty good, since at the time she was told it could be two weeks. After dinner she sat down to create her account. The first hurdle was creating a password. Banks have gotten nutty with password requirements, and Chase is no different in this regard. The password had to be 8 - 32 characters long, had to have a number and a capital letter, couldn’t have two or more letters in a sequence (e.g., abc), and couldn’t have more than two of the same letter.

This meant that trying to use a pass phrase, something NIST now suggests, wasn’t possible. The pass phrase she tried had three occurrences of the letter “a”. The web form didn’t display a message it just made the heading over the first password field red. Once we abandoned the pass phrase and used the Chrome suggested password we still couldn’t get the form to submit. Turns out that using auto fill for the phone number resulted in an unformatted number: 3335551212. The form wanted a formatted number: 333-555-1212. Really Chase? Really? It isn’t hard to do a little processing of the digits in the field to format the number for the customer. Again, the only hint the number was the problem was a red heading.

Using color as the only means of conveying information violates user accessibility. It’s also just plain rude. Eventually, after several frustrating attempts, after being made to feel stupid by not being able to fill in a simple form, the “Next” button was enabled.

At which point she was informed that access to the agreements was currently unavailable and she’d have to try again later. This is adding insult to injury. She wants to be a good customer. She wants to setup an account and play by the rules. Unfortunately the rules are obscure, poorly communicated, and seemingly arbitrary.

After signing out and signing back in, she was presented with the agreements, and she was able to digitally sign. Her account is now active and she can finally see a list of purchases.

To recap; the Apple Card application was simple and fast. It required a minimum of information. Approval was instantaneous and came with a fully functioning digital card. Within seconds of any purchase you get a notification on your phone showing you the merchant and the amount. Opening the card in the Wallet app lets you see spending by the week or month. You can see spending categories or spending by merchant. It. Just. Works. Oh, and you can pay any amount you want, at any time, toward your outstanding balance.

The Amazon VISA application process was fairly easy too. Where the process fell apart was having to wait a week to setup the account, and then having to jump through hoops to complete that process. I’ve been creating software for 4 decades now. I know what it takes to design and implement interfaces and back-end processes alike. Interfaces like the one at Chase rely too much on the developer and not enough on feedback driven design. Not providing messages to indicate when a field hasn’t been filled in properly, and relying only on a color change of the heading is a failure. Not formatting user inputs is a failure. Allowing the entire process to complete only to present the customer with a message saying, “Sorry. We can’t finish now because our systems aren’t available” is a failure.

My sincere hope is that the Apple Card is hugely successful. That its ease of use and clear concise reporting of expenditures, and the protection of customer information, sets an example that people begin to demand of other credit providers. Apple disrupted home computing with the Macintosh in 1984. They disrupted portable music, and indeed the music industry, with the iPod in 2001. They completely redefined cell phones with the iPhone release in 2007. Now I hope they are on the cusp of disrupting the credit industry.

If you have an iPhone and want a good credit card experience, go get an Apple Card.

Switching to Bitwarden


For the past eight or so years I’ve been using 1Password as my password manager. It works both on my Macs and on my iOS devices. The browser extension on the Macs has always been good, and the iOS one has gotten better. I like that it stores more than just passwords; email account, credit cards, passports, and secure notes to name a few. At last count I have nearly 600 total items stored in my 1Password vault.

Recently I’ve been exploring other password options. I have a laptop running Arch Linux, and a NUC that has Ubuntu 18 installed on it. I’d like to have password management there too. After a lot of research, and some experimentation, I’ve switched to Bitwarden.

Bitwarden is free, has a hosted or a self-hosted option, works on Windows, Linux, and macOS, and has browser extensions for Google Chrome, Mozilla Firefox, Safari, Vivaldi, Opera, Brave, Microsoft Edge, and even Tor Browser. There are mobile apps for iOS and Android, and there is a command line interface. Oh, and you can access your vault via the web as well.

The types of objects Bitwarden knows about out of the box include Logins, Cards, Identity, and Secure Notes. The Secure Notes object allows you to define new field to hold text, boolean values, or hidden text (think passwords). The interface is clean and lightweight.

While it is possible to export your 1Password vault and import it into Bitwarden, and even though I initially did just that, I decided to take a more curated approach. I deleted all the entries from the import and manually created entries for those items I wanted and frequently used in 1Password. Spread out over a couple of days it took me perhaps 4 hours to create 130 entries in my new Bitwarden vault. Along the way I made use Bitwarden’s password generator and cleaned up some weaker passwords, and some duplicate passwords.

On the 1Password side I added two “smart folders”: one called 1Password and one called Bitwarden. The 1Password folder displays all the items that do not have a tag with the value bitwarden. The Bitwarden folder does the opposite; it shows only items that do have a bitwarden tag. This allows me to know which 1Password entries I’ve moved.

Initially I setup the browser extension for Firefox. I use Firefox primary for work; it holds all the web applications I use as a part of my job. One option the extension has that is very nice is automatic fill. Open a page that has a recognizable login form and Bitwarden will pre-fill the user and password fields for you. All you need to do is press enter. Very nice.

Getting the Safari extension installed was more problematic. Every time I clicked on the link I was take to the Safari Extension Gallery, where there was no evidence of Bitwarden. After several attempts from different computers I finally did a search and found this issue on the Bitwarden GitHub account: Publish Safari Extensions as a Safari App Extensions. Toward the bottom of the comments there is a link to a new beta version of the desktop app that includes a working Safari extension. As Apple works to protect individual privacy they are changing underlying frameworks and altering long standing protocols, which in turn requires the developer put in extra effort in order to support what is likely a small(er) audience.

With the new extension in place, Safari (my primary browser) now auto-fills login forms and works as expected. I’ve also installed Bitwarden and the Firefox and Vivaldi extension on my Arch Linux laptop, and I’ve setup the iOS app and it’s extension on my iPhone and iPad.

I am pleased with Bitwarden, and I’m also pleased to have started reviewing and cleaning up my password vault. For now I’m keeping 1Password installed as I am sure there will be seldom used accounts that I’ll want to migrate. However, I’ve disabled the browser extensions for 1Password so that I’m only using the Bitwarden one.

Applications like Bitwarden and 1Password make good password hygiene possible and practical. If you aren’t using a password manager, I highly recommend either of these two applications.

TIL: Just Delete Me ⧁


A directory of direct links to delete your account from web services.

TIL What Does rc in Unix Mean? ⧁


Today I learned (TIL) what the rc in bashrc means.

What Computer to Buy?


I’d really like a new computer, but I want something that will meeting my current computer needs and have enough storage capacity to not only hold my current data, but enough to allow my current data to grow.

What I Currently Have

My current primary computer is a Late 2013 15" MacBook Pro with a Retina display, a 2.3 GHz Core i7 processor, 16 GB of RAM and a 1 TB SSD. Except for needing a battery replacement earlier this year (the original was starting to swell), and having a bad case of full storage, it has been and continues to be a superb machine. I’ve used it daily since getting it in June 2014 and it still does everything I want.

Being out of storage space, or nearly out of storage space, has been an issue for over a year now. Photos, music, audiobooks, videos, and documents, not to mention coding projects, applications and their data, all add up. Recently I was down to 30 GB free space. Some judicious moving of files to an external USB drive, and the deletion of old iOS backups, brought the free space figure up to ~ 150 GB.

I’ve already migrated all my audiobooks and movies to an older (a Mid 2009 MacBook Pro) laptop. I could use external drives to house my growing media collections, but carrying around a laptop with an attached external drive is nowhere near as useful and portable as a laptop by itself.

Here is the space used by all the directories in my home folder, along with the space used by the Applications folder.

Any new computer would need to have, at a minimum, 2 TB of storage. And that would only give me roughly 500 GB of room to grow. I have no idea how long it will take to fill 500 GB. It would be a nice cushion, but I fear it would be filled within a year’s time.

My current favorite configuration from Apple is a new 13" MacBook Pro, with Retina display, a 2.4 quad-core i5 processor, 16 GB of RAM, and a 2 TB SSD for $2,799. Moving up to a 2.8 GHz quad-core i7 adds $300 to the price, or $3,099.

An Aside on Annual Computing Cost

I bought my first laptop, A Titanium PowerBook G4 in 2002. In 2009 I upgraded to an aluminum MacBook Pro. And in 2014 I upgraded again to a Retina MacBook Pro. Combined these three computers cost roughly $8500. Divide 8500 by 17 years and you get an annual computer cost of $500 per year. A $3100 computer would need to last 6 years to keep that $500/year average.

Butterfly Keyboard Effect

My employment provides me with a 2017 13" MacBook Pro, with a 3.1 GHz core i5, 16 GB of RAM, and a 512 GB SSD. It’s a very nice machine; the size is perfect, the weight is subjectively considerably lighter than my 15", and with BetterTouchTool installed, the TouchBar is semi-useful. Until recently it hadn’t been plagued with butterfly keyboard switch issues. However, the “B” key now only works some of the time. Repeated presses with varying weight will eventually produce a “B”.

My fear of spending $3,099 on a new laptop only to have it develop issues with its keyboard is not significant, however, it is present. I’d be happier if I could get something without the butterfly keyboard. My work iMac has a Magic Keyboard that has good key travel and no keyboard issues.

Options, Options, Options

Using $3100 as an upper limit my question is: what computer can I buy that would:

  • hold my entire data collection (media, code, applications, and documents) in one place rather than on two separate computers
  • be enough computer to last another 5 years

If that computer happens to be a desktop, then would there enough left from the $3100 budget to replace my current laptop with something smaller and lighter? With a desktop for heavy lifting the future, smaller and lighter laptop (or iPad Pro?) needn’t be maxed out for CPU or storage.

Another option would be to invest money (and time and effort) into setting up a NAS with enough storage to hold the media collection (and enough room to expand it), and act as a backup target for all the computers in the house.

In the rest of this posting I want to explore these options:

  • Getting and setting up a 4 or 5-bay NAS with at least 12 TB of usable storage
  • Getting a Apple desktop with no less than 4 TB of storage (either with or without a lightweight laptop or iPad Pro). Storage could be external USB drives.
  • Getting a Apple laptop with no less and 2 TB of storage

NAS

A 4-bay Synology chassis can be bought for ~ $360. Four 6 TB Western Digital Red hard drives, at ~ $150 each, total another $600. So for a $960 investment I could have a NAS with 24 TB of raw capacity. A RAID 10 (mirror and stripe) configuration would net 12 TB of usable storage. That would easily swallow my media collections and provide back up for all the computers in the house.

A 5-bay Synology chassis is $899. Five 6 TB hard drives, at ~ $150 a piece, is $750, for a total cost of $1,649.

The 4-bay option is less expensive per terabyte of storage.

Playing media (music or a movie) across my home network would be acceptable. We currently use an Apple TV to view movies stored on my older MacBook Pro, and for the most part that works. What I don’t know, and would need to research, is could I access music or movies when I’m out of the house? And would that bump into the bandwidth consumption limits my ISP has?

Having a NAS - even without reliable media access while not at home - would be a huge improvement in reliable storage at home. It would also relive to pressure of having a constantly full hard drive. Having a NAS is likely something I should do regardless of getting or not getting a new computer.

Apple Desktop

Apple currently sells desktop computers at three price points. 27" iMacs, 27" iMac Pros, and Mac Minis. You can still get refurbished 2013 Mac Pros, but a new (to me) computer that is based on a thermally limited 6-year design does not make good sense. While I’d love to get one of the new Mac Pros being released later this year, I expect them to start at double my budget, and expect a nicely fitted out one to use up all of $10,000. Not an option unless some kind reader of zanshin.net wants to make a donation.

iMac

A 27" iMac, with 3.1 GHz 6-core i5 processor, 32 GB of RAM, and a 3TB Fusion drive, with Magic keyboard and a Magic Trackpad, sells for $2,949. I’d have $151 left over from my self-imposed budget. Spending all the money on a new desktop would preclude a new laptop or iPad for the time being.

An Aside on Fusion Drives

Apple’s fusion drive is a combination spinning platter drive and SSD. The OS optimizes how the drive is used. It is less expensive per GB, but I’m not sure I want a fusion drive. The 3 TB fusion drive adds $300 to the price. That same $300 will outfit the iMac with a 512 GB SSD. External USB drives would then be used to house all the data.

iMac Pro

A 27" iMac Pro, with a 3.2 GHz 8-core Xeon processor, 32 GB of RAM, and a 4 TB SSD, with a Magic keyboard and trackpad, sells for $6,249. More than twice the budget. Scratch the iMac Pro from consideration.

Mac Mini

A Mac Mini, with a 3.0 GHz 6-core i5 processor, 32 GB of RAM, and a 2 TB SSD, with no keyboard, no trackpad or mouse, and no monitor, sells for $2,499. It is possible to upgrade the RAM in a Mac Mini. 32 GB of after market RAM costs roughly $150. Getting only 8 GB of RAM reduces the price to $1,899.

A Magic keyboard and trackpad are another $228. I have a 24" monitor I could use, but a new 27" 5K monitor is ~ $550. Getting a Mac Mini that is equivalent to an 27" iMac costs $2,827. For that price you might as well get an iMac.

Approaching the Mac Mini from a different angle, and getting a minimal machine and adding RAM, and using external, directly connected storage (USB drives) looks like this: 3.0 GHz, 8 GB RAM, 512GB SSD for $1,299. Added RAM is $150. Two 4 TB USB drives are ~ $200. The keyboard and trackpad are still $228. And the monitor is still $550.

With either Option A or Option B, postponing the monitor purchase until a later date saves ~ $550, bring the two prices down to $1,877 and $2,277 respectively. At $2,277 a 2 TB Mac Mini is $672 less than the iMac. The $1,877 Mac Mini is $1,072 less.

Side-by-side, here is a minimal Mac Mini with 3rd party RAM, and external USB storage, next to an iMac with comparable RAM, and external storage.

Clearly a Mac Mini, with 3rd party RAM, and some external storage is the least expensive desktop option. The biggest difference between the two setups is the monitor. Apple makes simply gorgeous high resolution monitors. While there are some options for 4K or 5K monitors from 3rd-party manufacturers, those offerings are no where near as good looking as the iMac. And the LG monitor infamously had problems where it interfered with nearby WiFi access points.

The question becomes, is it worth an extra $723 to have the Apple display. One way to look at the iMac is, it’s the best 5K monitor money can buy, and it comes with a free computer glued to its backside.

Apple Laptop

Apple has three laptops in it’s current line up. The MacBook Air, and both a 13" and 15" MacBook Pro. Price and processing power, along with storage, vary widely.

MacBook Air

A MacBook Air, with a 1.6 GHz i5 processor, 16 GB of RAM, and a 512 GB SSD sells for $1,699. It would make a fantastic second computer. As a way out of my “need more storage” situation, it isn’t helpful.

MacBook Pro 13"

The 13" MacBook Pro, with a 2.4 GHz i5 processor, 16 GB of RAM, and a 2 TB SSD, sells for $2,799. If you select the largest processor option, a 2.8 GHz i7, you add $300, bringing the price up to $3,099. My benchmark machine, for better or for worse.

MacBook Pro 15"

All of my personal laptops have had some flavor of a 15" screen. A new 15" MacBook Pro with a 2.3 GHz processor, 32 GB of RAM, and a 2 TB SSD lists for $3,799. You can double the SSD to 4 TB for an additional $1,400, bringing the total up to $4,599. A little outside my budget. And a lot of money for a laptop.

What to Choose

The best choice, I feel, among the laptop options, is the 13" MacBook Pro with a 2 TB SSD and 16 GB of RAM. However, I don’t think that is the rational choice.

The rational choice is to buy a 4-bay NAS chassis and four 6 TB drives, and set that up. It eliminates the lack of storage problem, and provides much needed back up storage. For all the computers in the house. A NAS would be $960 well spent.

The “I want a shiny new toy” thing to do is to spend $3100 on an iMac and some external drives. The “still shiny but more affordable” thing to do is get a Mac Mini, a keyboard and trackpad, and a couple of external drives for ~ $1,876. Delay the monitor purchase hoping that Apple will release a standalone 5K monitor for a reasonable price in the next year or so.

In the end I think the first thing is to buy and setup a NAS with as much redundant storage that $1000 will buy. And wait to see what Apple does. There’s a rumored 16" MacBook (in the same form factor as today’s 15"). There is speculation about a 14" laptop to follow the 16". And I hope that sometime soon, Apple releases a new keyboard that eliminates once and for all, the problems with the butterfly switches.

Getting a NAS and waiting to see what Apple does is the smart move. With network storage I can offload quite a bit from my laptop, making it perform better. It still has life in it. However, if you want to donate to my cause, I’ll be happy to buy bigger, shinier toys from Apple.

My MacBook Pro Has A Swollen Battery


My late 2013 15" MacBook Pro Retina has a swollen battery. Recently I noticed that the trackpad was increasingly hard to click. In fact the upper half, the bit closest to the space bar on the keyboard, can’t be clicked at all. When you put the laptop on a flat surface it is starting to rock from side to side a little now too.

According to my Battery Health app, the current maximum charge possible is 7097 mAh out of 8440 mAh originally. Or 83% of its original capacity. The manufacture date is shown as May 16, 2014, so it is eight weeks shy of being 5 years old. It has seen 512 cycles. Apple suggests that my laptop battery has a useful life of 1,000 cycles.

Apple will replace my battery for $199. The nearest Apple store location is 110 miles away. There is a certified repair center about 15 miles away. Both require that you bring the laptop in for a diagnostic test first. This is ti “certify” what the issue is, so that genuine Apple OEM parts can be ordered. I was quoted 24-48 hours for the diagnostic, and 4-5 days for the battery shipment and replacement.

I have other computers (5 that I can reach from where I am sitting) but I really don’t like having my primary computer out of my hands for 5-7 days, or more.

A couple of searches led me to this YouTube video, which shows How to Replace a Late 2013 MacBook Pro Battery. After watching it a couple times, and after reading about the iFixIt kit used to make the repair, I wen ahead and ordered a [MacBook Pro 15" Retina (Late 2013-Mid 2014) Battery](https://www.ifixit.com/Store/Mac/MacBook-Pro-15-Inch-Retina-Late-2013-Mid-2014-Battery/IF117-046?o=5 “MacBook Pro 15” Retina Battery kit"). It should be here in 4 or 5 days.

I’ll post again with how the replacement went. Hopefully on this laptop with a new battery, and not on a hastily purchased new laptop.

Triple Boot Part 4: Install Ubuntu and Replace GRUB2 with systemd-boot


This posting is part of a multi-part series on configuring a laptop with three different operating systems using systemd-boot. The series starts with How to Install Three Operating Systems on One Laptop.

Like Windows 10, Ubuntu has straight forward installation process. The only deviation we are going to take from the default install is how we partition the hard drive. In the previous posting, while we setup Arch Linux, we prepared a partition for Ubuntu to use. Also, we want to map in the shared data partition.

Follow the guided install. When you reach the “Updates and other software” dialog you can make some choices. I choose the “Normal installation”, “Download updates while installed Ubuntu”, and “Install third-party software for graphics and Wi-Fi hardware and additional media formats” options.

On the “Installation type dialog, choose “Something else”. This will allow us to control exactly where Ubuntu gets installed. After clicking on “next” you will see a breakdown of the partitions on the drive.

Click /dev/sda6 to select it and then click “Change…”. In the popup dialog, change the “Use as:” drop down to be Ext4 journaling file system. And change the “Mount point:” to / (or root). Click “OK”.

Now click to select /dev/sda2 which should have “Windows Boot Manager” listed in the “System” column. This is the UEFI system partition that Windows created, and that we put systemd-boot into during the Arch Linux install. Ubuntu is going to put GRUB2 there. After the install is finished we will create a loader entry for Ubuntu and remove GRUB2.

With /dev/sda2 selected click “Change…”. The “Use as:” option should already say “EFI System Partition”. Click “OK”.

Click “Install Now” to continue the installation. After the installation process completes, and you reboot, you be presented with a GRUB2 boot loader, that will list Ubuntu, Arch, and Windows. If you are satisfied with that boot loader, then you are done. Congratulations, you have a machine with three operating systems coexisting, side-by-side.

Getting systemd-boot working with all three operating systems is a bit trickier. While the EFI specification (ESP) doesn’t prohibit having multiple ESP (EFI System Partition), it is not the best idea. A better idea is to have multiple subdirectories in your EFI partition, to separate individual operating systems from each other. After the Windows 10 install the EFI partition (in my setup mapped under Linux to /boot) has a single directory called EFI. This contains the Windows 10 boot loader. At the end of part 3 the loader.conf and loader/entires/arch.conf files were setup for Arch. The following kernel files were also created.

initramfs-linux-fallback.img
initramfs-linux.img
intel-ucode.img
vmlinuz-linux

Unless you are using multiple kernel versions with Arch Linux (something I’m not doing) there will always be a single kernel for Arch. Meaning the four files listed above will always be the only kernel files Arch depends upon.

Ubuntu does allow for the use of multiple kernels. These files all have a version number as a part of their name, e.g., vmlinuz-4.15.0-44-generic. While it would be possible to leave those files at /boot a tidier way to organize them is to create a directory for the Ubuntu systemd-boot files. Rather than have to manually update the files, and associated configuration changes with each kernel update or change, kernel hooks can be used to automatically move the kernel files to the proper directory and update the loader/entires files.

Setup systemd-boot for Ubuntu

I followed Replace GRUB2 with systemd-boot on Ubuntu 18.04 and the Ubuntu 16.04 systemd-boot gist to configure my existing systemd-boot setup to include the newly installed Ubuntu.

As the blobfolio.com posting points out,

Unlike a lot of software, boot loaders can generally exist independently of one another. To be safe, you should leave your working GRUB2 around until you are confident you have systemd-boot set up correctly. If you screw something up, don’t sweat it! Just re-reboot, open your BIOS’ boot menu, and point it to GRUB2.

In the /boot directory you will find all the kernels and related files that Ubuntu installed. Debian-based systems like Ubuntu won’t install package files to a FAT partition. Since the /boot/efi directory is really a FAT32 file system mount point, none of the Ubuntu kernels will be installed there. You’ll have to copy them yourself.

sudo -i   # Run these commands as root, it's easier
cd /boot/efi
mkdir ubuntu
cd ubuntu
cp config-* /boot/efi/ubuntu
cp initrd.img-* /boot/efi/ubuntu
cp System.map-* /boot/efi/ubuntu
cp vmlinuz-* /boot/efi/ubuntu

With a directory under /boot/efi just for Ubuntu it will be easier to keep kernels for different distributions tidy. My 18.04 installation dropped two kernels, 4.15.0-29 and 4.15.0.43, so there were two files each for config-*, initrd.img-*, System.map-*, and vmlinuz-*. With the files copied it’s time to setup a loader entry for Ubuntu.

Switch to the /boot/efi/loader/entries directory and create a new *.conf file for each Ubuntu kernel you want to expose in the boot loader list. I setup two files, ubuntu-4.15.0-29-generic.conf and ubuntu-4.15.0-43-generic.conf.

cd /boot/efi/loader/entires
vi ubuntu-4.15.0-29-generic.conf
cp ubuntu-4.15.0-29-generic.conf ubuntu-4.15.0-43-generic.conf
vi ubuntu-4.15.0-43-generic.conf

Both files look like this:

title   Ubuntu 4.15.0-##-generic
linux   /ubuntu/vmlinuz-4.15.0-##-generic
initrd  /ubuntu/initrd.img-4.15.0-##-generic
options root=PARTUUI=26032c35-9f53-fd41-93a2-9bb466e572cf rw

Where ## is either 43 or 29. As with the configuration file for Arch Linux, lookup the PARTUUID for the Ubuntu root partition (/dev/sda6) with

blkid -s PARTUUID -o value /dev/sda6

Since Ubuntu is a graphical environment, you can do this in a separate terminal and copy the result into your conf file.

I used a slightly modified version of the script giving in the blobfolio article. Here’s my script:

# This is a simple kernel hook to populate the systemd-boot entries
# whenever kernels are added or removed.
#

# The PARTUUID of your disk.
# Must be PARUUID and not UUID
PARTUUID="9dad7c53-d6d2-754e-8896-26feff826adf"
ROOTFLAGS="errors=remount-ro"

# Our kernels.
KERNELS=()
FIND="find /boot -maxdepth 1 -name 'vmlinuz-*' -type f -print0 | sort -rz"
while IFS= read -r -u3 -d $'\0' LINE; do
	KERNEL=$(basename "${LINE}")
	KERNELS+=("${KERNEL:8}")
done 3< <(eval "${FIND}")

# There has to be at least one kernel.
if [ ${#KERNELS[@]} -lt 1 ]; then
	echo -e "\e[2msystemd-boot\e[0m \e[1;31mNo kernels found.\e[0m"
	exit 1
fi

# Perform a nuclear clean to ensure everything is always in perfect
# sync.
rm /boot/loader/entries/ubuntu*.conf
rm -rf /boot/efi/ubuntu
mkdir /boot/efi/ubuntu

# Copy the latest kernel files to a consistent place so we can keep
# using the same loader configuration.
LATEST="${KERNELS[@]:0:1}"
echo -e "\e[2msystemd-boot\e[0m \e[1;32m${LATEST}\e[0m"
for FILE in config initrd.img System.map vmlinuz; do
    cp "/boot/${FILE}-${LATEST}" "/boot/efi/ubuntu/${FILE}"
    cat << EOF > /boot/efi/loader/entries/ubuntu.conf
title   Ubuntu ${LATEST}
linux   /ubuntu/vmlinuz
initrd  /ubuntu/initrd.img
options root=PARTUUID=${PARTUUID} rw rootflags=${ROOTFLAGS}
EOF
done

# Copy any legacy kernels over too, but maintain their version-based
# names to avoid collisions.
if [ ${#KERNELS[@]} -gt 1 ]; then
	LEGACY=("${KERNELS[@]:1}")
	for VERSION in "${LEGACY[@]}"; do
	    echo -e "\e[2msystemd-boot\e[0m \e[1;32m${VERSION}\e[0m"
	    for FILE in config initrd.img System.map vmlinuz; do
	        cp "/boot/${FILE}-${VERSION}" "/boot/efi/ubuntu/${FILE}-${VERSION}"
	        cat << EOF > /boot/efi/loader/entries/ubuntu-${VERSION}.conf
title   Ubuntu ${VERSION}
linux   /ubuntu/vmlinuz-${VERSION}
initrd  /ubuntu/initrd.img-${VERSION}
options root=${PARTUUID} rw rootflags=${ROOTFLAGS}
EOF
	    done
	done
fi

# Success!
exit 0

The script makes sure there is a kernel present to work with. Then it removes all the ubuntu*-conf files from /boot/loader/entires, deletes the /boot/efi/ubuntu directory and recreates it. I.e., empties it. It then makes a new ubuntu.conf file for the latest kernel version present and copies those files into place. Next it makes ubuntu-#.##.#-##-generic.conf file for all other kernels present, and moves their files into place.

The script is executed by two kernel hooks. The postinst (post install) and postrm (post remove) hooks. The hooks are directories in /etc/kernel. With the script copied into place and given the proper ownership and permissions everything is set.

During the next apt upgrade that includes kernel changes the script will be run automatically. It outputs messages to the console so you can see what it is doing. (I have some typos in my script, and had to run it a couple times to get it working.

With everything in place, reboot, and change the boot order to put the Linux Boot Manager first. This is the entry that corresponds to the Arch systemd-boot configuration. On my BIOS the ubuntu entry is second. This corresponds to the GRUB2 setup that Ubuntu installed. The last entry is Windows Boot Manager which is, of course, the Windows boot manager.

Hopefully the boot manager will appear listing Arch Linux, one or more Ubuntu instances, and the Windows boot Manager. Test each entry to make sure they all work. I misspelled vmlinuz as vmlinux and got an error when trying to load Ubuntu.

Once everything is tested you can, if you like, remove the GRUB2 setup from Ubuntu.

Purge the package:

sudo apt purge grub*

And purge any obsolete dependencies:

apt autoremove --purge

This may or may not remove all the files GRUB deposited into /boot and /boot/efi. You can manually remove any leftovers.

Summary

You should now have systemd-boot working with three different operating systems, Windows 10, Arch Linux, and Ubuntu 18 LTS. Admittedly, setting up systemd-boot with three operating systems is a lot of work. That GRUB2 does it with little or no fuss is perhaps a compelling reason to stick with GRUB. However, GRUB is getting a bit long in the tooth. systemd-boot is one alternative. rEFInd is another alternative.

If you have followed some or all of this series drop me an email and let me know how it turned out.

This posting is part of a mulit-part series on installing three operating systems on a single laptop.

Triple Boot Part 3: Install Arch Linux and Setup systemd-boot


This posting is part of a multi-part series on configuring a laptop with three different operating systems using systemd-boot. The series starts with How to Install Three Operating Systems on One Laptop.

Unlike most Linux distributions, Arch Linux doesn’t have an installer. Instead you are dropped into a live Arch environment, running off the install media, that allows you to create your Arch installation from the disk up. The first time installing Arch can be a bit daunting, but after you make a few mistakes and start over once or twice, it’ll start to make more sense.

The Arch Linux Wiki is vast and comprehensive. It is an excellent reference for any Linux distribution, not just Arch. Like a lot of documentation, however, it is sometimes a bit lacking in actual examples. Read the documentation carefully and completely and you should be okay.

It is best to follow the Installation Guide. I like to augment the guide using a couple of YouTube channels.

LearnLinuz.tv has great content including several multi-part series on installing Arch Linux. The Getting Started with Arch Linux (3rd Edition) series is the latest, and well worth a watch. Jay, the host, walks you through installing Arch, setting up pacman, installing a desktop environment, and more.

GloriousEggroll also has some excellent video tutorials. I used his Arch Linux NetworkManager / Wifi Setup Guide to solve issues I had recently getting network connectivity after the first boot into a new Arch installation.

There are dozens and dozens of YouTube channels and videos dedicated to Arch Linux, some are better than others.

Here are the steps I followed to add Arch Linux as the second of three operating systems on my Asus.

Step 1

Create bootable USB with Arch Linux ISO

Step 2

Boot from installer USB. The resulting screen will look like this:

Arch Linux 4.20.0-arch1-1-ARCH (tty1)

archiso login: root (automatic login)

root@archiso ~ #

Step 3

Use wifi-menu to attach to network and verify you have Internet connectivity

wifi-menu
ping google.com

On my Asus there is a short delay after running wifi-menu before the ping runs successfully.

Step 4

Install tmux

pacman -Sy tmux

I find it useful to use tmux when it is time to chroot in to the new Arch installation. I’ll explain more then.

Step 5

Verify boot mode

ls /sys/firmware/efi/efivars

If nothing lists then you aren’t running under UEFI. Stop and go sort that out at your computers BIOS setup screen.

Step 6

Update system clock

timedatectl set-ntp true

Step 7

Partition the disk to add the partitions both the Arch and Ubuntu installations will use, as well as the shared data partition. My SSD has a total capacity of 500 GB, and approximately 120 of those were used for Windows 10. The sizes below are based on those figures. Your sizes may vary. Void where prohibited.

fdisk -l

Note that /dev/sda1-4 are used by Windows, we want to leave these in place. Now run fdisk against the drive to partition it.

fdisk /dev/sda
  • Create /dev/sda5 as 60G Linux file system. This will hold root and home for Arch.
  • Create /dev/sda6 as 60G Linux file system. This will be root and home for Ubuntu.
  • Create /dev/sda7 as ~240G Linux file system. This will be the shared data partition.

Step 8

Formatted the newly created partitions to be Linux file systems.

mkfs.ext4 /dev/sda5
mkfs.ext4 /dev/sda6
mkfs.ext4 /dev/sda7

Step 9

Mount the partitions so we can access them. In my setup /dev/sda5 is the root+home partition for Arch. /dev/sda2 is the UEFI partition created when Windows 10 was installed. Make sure to use your partitions numbers here.

mount /dev/sda5 /mnt
mkdir /mnt/boot
mount /dev/sda2 /mnt/boot   # this is the EFI partition Windows 10 created

Step 10

Run the install. The step you’ve been waiting for.

pacstrap -i /mnt base base-devel

Say yes to all members in both the base and base-devel packages. I selected the systemd-resolved option. Depending on the speed of your Internet connection this make take a little while. Sit back, relax. Enjoy a Mexican Coca-Cola with real sugar.

Step 11

Mount the data partition. In the next step we are going to generate the fstab (file system table) and we want this drive mapping included.

mkdir /mnt/data
mount /dev/sda7 /mhn/data

Step 12

Generate the file system table (fstab).

genfstab -U -p /mnt >> /mnt/etc/fstab

Step 13

Change root into the installed system using tmux. The chroot command changes the apparent root of the file system. But running it you are in effect running your newly install Arch instance. By using tmux you can have a second screen, which may be useful in looking up or displaying information. UUIDs or PARTUUIDs, for example. tmmux is relatively easy to use. The tmux command will create a new session. Inside that session ctrl+b % will split the screen vertically and ctrl+b rightarrow or ctrl+b leftarrow will switch between the two screens. Typing exit will exit the split screen or the session.

tmux    # ctrl+b % to split ctrl+arrow to move
arch-chroot /mnt

Step 14

Install more things. Not all of these packages are necessary to complete a basic Arch install, but I’m going to want them eventually so why not install them now.

pacman -Sy openssh linux-headers git neovim vim tmux wpa_supplicant networkmanager

Step 15

Set time zone and hardware clock. I live in the Central time zone of the United States. Adjust the region and city to your time zone. You can list the directory contents of /usr/share/zoneinfo to find what you need.

ln -sf  /usr/share/zoneinfo/America/Chicago /etc/localtime
hwclock --systohc --utc

Step 16

Set your locale. Localization is used by programs and services in Linux.

vi /etc/locale.gen   # Uncomment en_US.UTF-8

If you aren’t familiar with vi or vim use nano. Or learn some vi basics. It’s on virtually every Linux-based system you’ll ever interact with.

Now generate the locale.

locale-gen

Step 17

Set a root password. Pick a good one and remember it. If you do forget it, you can use the installer to mount your partitons, chroot into the Arch install and change the password again.

passwd

Step 18

Setup systemd-boot. I will admit that I have this working, and I understand what I did, but I am not a systemd expert nor am I a systemd-boot expert. I botched this on one earlier install attempt (had the wrong partition (recovery instead of EFI) mapped and ended up starting over.

bootctl --path /boot install
bootctl update

Following the documentation on the systemd-boot page, the /boot/loader/loader.conf file needs to be edited. Following that a “loader entry” for Arch needs to be created. In addition to the Arch wiki page I found Installing Arch Linux the EFI/systemd-boot Way helpful.

Edit /boot/loader/loader.conf. Mine looks like this:

default arch
timeout 5
editor  no

The default arch line indicates the loader entry is called arch.conf (the .conf suffix is not required). The boot loader will wait 5 seconds before booting the default OS. And finally, the editor setting determines whether one can access the kernel parameters editor or not. Since the root password can be bypassed, no is the suggested value.

Now create /boot/loader/entires/arch.conf. There is a basic sample at /usr/share/systemd/bootctl/loader.conf.

Mine looks like this:

title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options root=PARTUUID=58004e6b-c0ae-134f-bc93-c319667025f5 rw

In order to get the PARTUUID for the root partition, run this command:

blkid -s PARTUUID -o value /dev/sda5

Note: PARTUUID is not the same as UUID.

The intel-ucode.img file named in that configuration had to be installed on my setup.

pacman -S intel-ucode

More about Microcode.

After all the above steps if you list the directory at /boot you should see something like this:

EFI initramfs-linux-fallback.img initramfs-linux.img intel-ucode.img loader vmlinuz-linux

EFI and loader are directories. loader should have your loader.conf file and a sub-directory called entries that has your arch.conf file.

Step 19

Exit from the chroot (and tmux if you used it) and unmount all your mounts and reboot. This is the last step. When your system reboots you should be at a login prompt in your Arch installation.

exit
umount -R /mnt
reboot

When the machine reboots you should see the boot loader screen (if you selected a timeout value in your arch.conf) with entires for “Arch Linux”, “Windows Boot Manger”, and “Reboot into Firmware Interface”.

If you aren’t at a login prompt, backtrack through the steps, and look up any error messages presented. Good luck.

Post Install Setup

Briefly here are a few things I do immediately after Arch boot successfully.

Finish locale setup

It wasn’t possible to run the localectl command prior to the instance booting.

localectl set-locale LANG="en_US.UTF-8"

Set a hostname

hostnamectl set-hostname dvorak

Add an entry to /etc/hosts

127.0.0.1   dvorak

Setup network access from command line

This can be tricky. Depending on the chipset your network interface(s) these directions may or may not be of use.

Running lspci -k will output all your kernel drivers. Find your wireless driver (or Ethernet) and use the driver name, iwlwifi e.g., to search for documentation on proper setup.

The steps below ensure that only NetworkManager is running and only it is controller the network interfaces.

Disable netctl

Lookup the name of the wireless interface

ip a

Wireless interfaces start with wl. Mine is called wlp1s0. You may or may not have netctl enabled. This following command will disable it, if it is present.

systemctl disable netctl-auto@wlp1s0.service

Enable NetworkManager

Run the following systemctl command to enable NetworkManger. The capitalization is important.

systemctl enable NetworkManager.service

You should see some output indicating the several symlinks were created. To complete the NetworkManager setup you need to reboot.

reboot

Use nmcli tool to setup connection

NetworkManger has a command line tool called nmcli. Run the following commands to get a list of access points and then connect to the AP of your choice.

nmcli device wifi list   # get a list of wifi access points
nmcli device wifi connect <BSSID> password <password>
ip a

The ip a command at the end should show an IP address assigned to your wireless network interface.

Create a user account for yourself

If you installed base-devel way back on the pacstrap command, the sudo package is already installed. Use visudo to enable the wheel group. Search the file and uncomment the wheel section.

visudo

Create your account.

useradd -mg users -G wheel,storage,power -s /bin/bash <accountname>

Set your password.

passwd <accountname>

Create a swapfile

I like to use a swapfile over a swap partition. The file is easier to resize.

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
echo '/swapfile none swap sw 0 0' | tee -a /etc/fstab

The statements above create a 2G swapfile at /swapfile, change the permissions, format it as swap, and append its definition to the existing fstab.

Summary

Hopefully you now have a working, if bare bones, Arch Linux installation. You should also have a working systemd boot loader that includes both Arch and Windows, and that lets you boot into either OS. In the next posting I’ll add Ubuntu 18 to the mix, and replace the Grub boot loader it installs with systemd-boot. Proceed to Triple Boot Part 4: Install Ubuntu 18 Desktop and Replace GRUB with systemd-boot.

This posting is part of a mulit-part series on installing three operating systems on a single laptop.

Triple Boot Part 2: Install Windows 10 with a Larger EFI Partition


This posting is part of a multi-part series on configuring a laptop with three different operating systems using systemd-boot. The series starts with How to Install Three Operating Systems on One Laptop.

Installing Windows is very straight forward; answer a few questions (and mute Cortana) and you’re all set. However if you want to alter the size of the EFI partition to be larger the process is not so straight forward.

I was able to figure it out with the help of this article: How to Change the UEFI System Partiton Size in Windows Setup.

In a nutshell when the partition dialog appears you will open a command line window, and use the diskpart command to delete everything but the Recovery partition. Next you create and format a larger EFI partition. When you leave the command line and return to the graphical dialog and reselect New the installer will create the missing partitions and keep the ones already created.

From the article:

  • Select your installation target and make sure it has no partitions (except unallocated space)
  • Click the New and then the Apply button.

You should now have four partitions: Recovery, System (ESP), MSR, and Primary.

  • Select each of the System, MSR, and Primary partitions in turn and click the Delete button to delete these partitions. Leave the Recovery partition in place.
  • Press Shift+F10 to open the Command Prompt
  • Type diskpart.exe and press Enter to open the disk partitioning tool
  • Type list disk and press Enter to list out your disks
  • Type select disk n where n is the number for the disk you want to install to as identified by the above command and press Enter
  • Type create partition efi size=550 where 550 is the desired size of the ESP in Mebibytes (MiB), and press Enter
  • Type format quick fs=fat32 label=System and press Enter to format the ESP
  • Type exit and press Enter to exit the disk partitioning tool
  • Type exit and press Enter again to exit the Command Prompt

You should now be back in the graphical Windows Setup partitioning tool where nothing has changed since the last time you looked at it.

  • Click the Refresh button to detect your partition changes

You should now have a disk with a default Windows Recovery tools partition, a 500 MiB UEFI System Partition, and some unallocated space for your Windows installation.

Select the unallocated space from the disk list and click the New button to automatically recreate the MSR and System partition in the remaining space.

Proceed with the rest of the install following the prompts. Mute Cortana or suffer the results.

Once the Windows installation is finished you are ready to move on to Triple Boot Part 3: Install Arch Linux and Set Up systemd-boot.

This posting is part of a mulit-part series on installing three operating systems on a single laptop.