Apple creates great experiences, but migrating to a new computer tends to not be one of them. There are several different ways of migrating, it’s not clear what the relative differences are between them, and when it’s taking a lot longer than you expect it’s not clear why. The seemingly never-ending migration hits programmers especially hard.
After running into this problem yet again recently, I did some research and experimented until I figured out what was wrong and how to fix it. Here’s what I learned.
Target Disk Mode
You may have gotten used to booting your old computer into Target Disk Mode and then having the new computer migrate the contents of the old hard drive over during setup. Don’t do that! I don’t exactly understand why, but it seems that Target Disk Mode can’t transfer at Thunderbolt speeds; it just degrades to regular USB transfer speed.
Migration Assistant
Instead of Target Disk Mode use Migration Assistant. Migration Assistant is an application you run on your old computer. It will restart into a special mode that is complementary to the setup mode your new computer boots into and will allow your new computer to transfer files from your old computer at Thunderbolt speeds.
Thunderbolt cable
Make sure you have a Thunderbolt cable. The situation with USB-C and Thunderbolt is confusing. The ports look identical and the cables almost look identical. The charging cable that comes with your Mac is not a Thunderbolt cable. You’ll know if you have a Thunderbolt cable because it will have a Thunderbolt symbol on at least one end of the cable.
The transfer can also be done over WiFi for convenience if you don’t have much data to migrate, so Migration Assistant will ask you to connect to your WiFi network, but it’s not very clear why it’s asking. It’s extremely easy to misunderstand what is happening and connect to your WiFi network even though you have no intention of migrating your data that way. If you then also mistakenly use a USB-C cable instead of a Thunderbolt cable, you can wind up starting the transfer thinking it’s using Thunderbolt when it’s actually using WiFi.
Thunderbolt ports
Not all USB-C ports on all MacBooks are Thunderbolt ports. I believe the 2016 and 2017 13-inch MacBook Pros without Touch Bar only support full-speed data transfer on the left two ports. You can’t use the right two ports. The 2016 and 2017 13-inch MacBook Pros with Touch Bar and the 2016 and 2017 15-inch MacBook Pros all have full-speed data transfer on all four ports. It’s really confusing. Thankfully, all the 2018 MacBook Pros with Touch Bar now support full-speed data transfer on all four ports.
Zip your code folder
The final hurdle is the files themselves. I noticed that the transfer speed would fluctuate all over the place. Sometimes it would spike up to Thunderbolt speeds but often it would slow to a crawl, causing the transfer to take hours longer than would be expected.
Eventually I realized that even though I only have about 400 GB of data to transfer, the reason why it takes so much longer than expected is because I’m a programmer.
Migration Assistant does a file-level copy—it copies each file one at a time. There’s overhead in starting to copy a file and finishing copying a file. That means it’s much faster to copy one big file than a lot of little files, even if in aggregate they’re the same size as the big file.
The work programmers do doesn’t take up a lot of space so it deceptively seems like the data should transfer quickly, but the package managers we use to install dependencies generally create a lot of files. I just looked at one project and the dependencies totaled ~120,000 files. Multiply that across 20 or 30 projects and that is what makes the transfer take so long.
The fix is to zip up the folder where you keep your code, delete the original folder, do the transfer, then unzip it afterwards.
Before I got around 5 MB/s when migrating. Sometimes it would spike up to 20 MB/s but only briefly. I don’t even know how long the transfer would have ultimately taken because I cut it off after 20 hours with only about two-thirds done.
I made another attempt after figuring all this out. This time the transfer started at 5 MB/s but slowly ramped up to 105 MB/s and held steady there for the rest of the migration.
It took 1 hour and 45 minutes. 🚀
Top comments (12)
Why not just use a Time Machine backup?
That could work! I believe Time Machine might do a block-level copy when restoring? If that’s the case it would obviate the need to zip your code folder. But not everyone has the gear to do a Time Machine backup. And many people that do use Time Machine use a Time Capsule or some other NAS (network-attached storage) and do their backups over their network to avoid mounting/ejecting every time they take their laptop. You wouldn’t be able to quickly restore an entire hard drive over the network and whatever networked drive you have probably doesn’t have a Thunderbolt, so you’re right back to the slow lane. 😞
No TM does file level incremental copying. It keeps tracks of the files that changed, copies does and does hard links of the other ones. AFAIK it doesn't use block copying at all. It needs to know about files, because you can tell TM to restore a single file back in time. Hence the name.
In my use case it's okay because I seldom have to restore to a new computer.
Another possibility is to just clone your hard disk with stuff like Carbon Copy Cloner which used to support block-level copy. The issue though is that the backup part is going to be slower, because you can't use incremental copy so you need to backup the entire disk everytime.
Anyhow 105 MB/s is not that much. 10 GB Ethernet, USB 3 and all Thunderbolts are faster than that. Are you sure you didn't have any other issues?
Oh, I knew Time Machine did file-level copying for incremental backups, I thought for a moment it might do block-level when restoring, but now that I think about it I don’t think that’s actually possible. In that case, I’m unclear why you’re suggesting migrating using a Time Machine backup rather than computer to computer? Is there some benefit I’m not aware of?
Yes, it’s almost definitely faster to do a block-level copy between the old computer and the new computer, however that requires you to have a third computer, which most people don’t have. I didn’t necessarily explicitly say this anywhere in the post, but the explanation was targeted at people who are doing the normal migration path, but it’s slow and aren't sure why or whether there is something simple they can do to fix it.
Regarding the speed, it’s not quite that simple… Gigabit ethernet is 1 Gbps, USB 3.1 is 10 Gbps, and Thunderbolt is 40 Gbps. “Gbps” is “gigabits per second” (not gigabytes per second). There are 8 bits in 1 byte, so Thunderbolt can theoretically do 5 GB/s or 5,000 MB/s. But the real bottleneck nowadays is the hard drive. The best SSDs can write at ~500 MB/s, but most SSDs write at around 300–400 MB/s, and that’s the theoretical maximums. Doing a file-level copy you’re going to have more overheard, so 105 MB/s seems fairly reasonable to me.
I suggested using TM because in an ideal scenario you should already have a backup of your primary computer's hard drive anyway. So you attach such backup (being through a network or a local USB drive) to a computer and start the restore. MacOS even asks you if you want to restore from a backup after installing the os.
It's true that transferring a single zip file is faster but my hard disk is full of images, videos, source code and documents.
It's going to take hours to zip all of that (and I don't really have the free space to hold such zip file on the same drive).
In my case zipping will take hours, plus transferring, plus unzipping which is the still going to take considerable time.
That's why I'm better off with time machine, just a different scenario I guess
I think I must still be missing something here… my understanding is that restoring via another Mac and restoring via Time Machine backup (assuming the backup is on an SSD) are effectively identical. You seem to be saying that you’re better off restoring via a Time Machine backup than zipping up your code folder, but I don’t understand why? What is different about restoring via a Time Machine backup?
You already have a backup in this scenario so you just have to restore to the new computer. No zipping time.
Right, I get what you’re saying, but I don’t understand why? Why would restoring from a Time Machine backup obviate the need to zip up the code folder? Why would restoring from a Time Machine backup happen more quickly than restoring from another Mac?
Zipping and unzipping also takes time, my current "dev" folder is 8 GB, around 425 thousand files.
Restoring from TM might take less time because you don't have the time you spend zipping and unzipping a giant folder.
Your post is about migrating from another Mac, I suggested that if someone already has a backup of that Mac, attaching the backup to the new machine might take less time, given that you don't just have only the "dev" folder, but at least your entire
$HOMEif you want everything to work correctly.So, in your scenario you still have to use migration assistant to copy the files minus the dev folder which will be transfered zipped. So you add the time to transfer everything except the dev folder, the time to zip the dev folder in the first place and the time to unzip it in the second computer.
I was just proposing that in some scenarios leaving the dev folder unzipped and let TM restore that it might take more or less the same time, that's it :)
OK, I think I see where the confusion is—I thought you were saying that there was something special you knew about Time Machine backups that allowed them to somehow be migrated/restored faster.
Migrating from a Mac and migrating from a Time Machine backup (assuming the backup is on the same type of hard drive as the Mac) are effectively equivalent. They should take the exact same amount of time. In my case, that was at least 20+ hours because of the overhead of doing a file-level copy on so many files.
My dev folder is ~9 GB and ~340,000 files. It took 15 minutes to zip it and ~3 minutes to unzip it.
With a zipped dev folder, migrating from another Mac took 1 hour and 45 minutes. That means it definitely doesn’t make sense to migrate using Time Machine. It’s not even close… the time it takes to zip and unzip the dev folder is tiny compared to how much time you save during the migration.
Ok got it. My experience of a full restore from a TM backup is of a few hours so I never had to consider a plan B. I'll keep that in mind if I have to
👍🏼 Yeah, I’ve heard wildly different numbers on how long migrations take. I know some programmers for who it only takes a few hours and I know other programmers like myself for who it takes a whole day. 🤷🏼♂️