You are not logged in.

Announcement

[2017.09.08] DeployStudio build v1.7.8 (checksum, release note).
[2016.08.26] DeployStudio build v1.6.19 (release note).
[2013.02.23] DeployStudio last universal build v1.5.17 (release note).

#1 2019-10-08 17:08:24

jelockwood
Member
Registered: 2009-11-11

DeployStudio, Mojave and Catalina

Firstly an update on my own experiences with Mojave.

1. It is not possible to NetBoot new T2 Macs under any circumstances, we can be confident Apple are not going to restore this capability so Macs from roughly 2018 onwards cannot be netbooted - basically any Macs with a T2 chip
2. It is not with the current out of date DeployStudio possible to build a DeployStudio Runtime boot drive that matches Mojave let alone Catalina, the newest DeployStudio Runtime drive you can make is High Sierra 10.13.6, this can still be useful
3. You can make a USB based Mojave boot drive which contains the full Mojave installation and then put a copy of the DeployStudio Runtime app on that, you can then run that app and connect to the DeployStudio repository, this can then let you run workflows

Based on the above for pre T2 Macs we netboot, then we erase the drive, then we run a workflow which runs a script which runs the standard 'startosinstall' command. For T2 Macs we boot from a USB drive containing a full Mojave installation and run the DeployStudio Runtime app and then do the same thing. After Mojave is installed we immediately reboot before completing the Apple Wizard - booting in the same pre/post T2 method and then run another DeployStudio workflow which does settings, pkg installations etc.

Note: This means turning off SecureBoot on T2 Macs - at least whilst doing this.


For Catalina as one might expect things have changed and more things are broken.

I have managed to create a Catalina installer and script, this can be run on pre T2 Macs via netboot like Mojave as above. For T2 Macs - which I have not yet tried I would expect to need to boot from a USB drive with a full Mojave or Catalina OS depending on the model of Mac. However I did find one major Catalina issue. Before I cover that I have been able to netboot a 2017 model MacBook Pro, run a script and install Catalina.

What I have now found with Catalina is that when running my main DeployStudio workflow which would do settings and pkg installs etc. this fails. It fails right at the beginning. This is because the NetBoot image is a High Sierra based image and whilst the MacBook Pro in question will happily boot from this High Sierra cannot access important parts of the Catalina drive. Remember Catalina makes significant changes to the drive structure and it seems this includes issues above and beyond the fact the data (user) space is now on a separate volume.

It turns out that High Sierra and Mojave cannot see the content of /private on a Catalina boot drive. As a result the initial DeployStudio runtime script immediately fails because it is trying to check the content of directories inside /private

To progress further my plan is to make a new Catalina based USB boot drive and again put a copy of the DeployStudio Runtime app on that. If that runs successfully I will then try running my main workflow again.

I have confirmed that Terminal in Mojave cannot see the Catalina contents of /private and that Terminal in the Catalina recovery partition _can_ see the contents of the Catalina /private directory. I therefore see no reason why DeployStudio runtime script should not be able to as long as it otherwise works in Catalina which is going to be my next task once I get and build a new USB drive.


Note: I have long given up on restoring 'fat' images. I have also given up on using AutoDmg images although I believe this would be possible for Mojave at least.

Offline

#2 2019-10-10 20:07:55

sebus
Member
Registered: 2011-07-19

Re: DeployStudio, Mojave and Catalina

Do we at any point capture Mojave image (to be able to just do restore on another machine?

Offline

#3 2019-10-10 20:30:11

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

> sebus wrote:

> Do we at any point capture Mojave image (to be able to just do restore on another machine?

As I mentioned I long ago gave up on using restoring 'fat' images. For Mojave I have successfully used AutoDmg but these days I use either a bootable macOS install disk or run a script which in turn runs the startosinstall command.

It may be possible to do a fat image restore but this approach does not run any firmware updates and also does not correctly add and name network interfaces. Since almost every update to Mojave has included firmware updates this is an important issue and a major fact as to why Apple actively disavow using imaging any more.

I have ordered a new bigger USB drive which I will therefore use to build a full Catalina operating system on along with the DeployStudio Runtime app and hopefully this approach will still work to run the script which runs the startosinstall and also then the main Deploystudio workflow to configure things.

Offline

#4 2019-10-11 22:23:01

sebus
Member
Registered: 2011-07-19

Re: DeployStudio, Mojave and Catalina

Sure, I can boot with USB-C SSD (which is almost as fast as internal drive on MBP 15,4) to  full install of Mojave (actually a clone of internal drive at some point, before I blown it away few times)
I can run DS Runtime, so I suppose I can run few bits that I need (rename, few little packages installs etc)

What I miss from my "fat" image is pre-configured default user folder

AutoDmg images work perfectly fine for asr restore in Mojave, in which case there is no need to reboot to carry on with DS settings etc

Last edited by sebus (2019-10-11 22:28:47)

Offline

#5 2019-10-12 09:15:54

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

The way I preconfigure the default user folder is a combination of a script running 'default write' commands and also copying some pre-configured files in to the appropriate places.

Also somethings that would be configured this way might be configurable via MDM settings.

Offline

#6 2019-10-12 15:19:22

sebus
Member
Registered: 2011-07-19

Re: DeployStudio, Mojave and Catalina

I am sure it can be configured with MDM, I still like the simplicity of DS to do things
I see no reason why DS can not be used together with MDM

Offline

#7 2019-10-14 12:31:05

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

Ok I got my new USB drive and have made some tests trying to use DeployStudio with Catalina.

As already mentioned I can have a Catalina boot disk and on that have the DeployStudio Runtime and via that login to the DeployStudio repo. I can also run a workflow which runs a script which does nothing except run the standard apple startosinstall command and this successfully triggers the install in this case of Catalina.

What I was then trying to do was run the rest of my normal configuration workflow. This time because I am running DeployStudio Runtime from within a Catalina version of the operating system it does see the contents of /private and hence can see /private/var, /private/var/db and /private/etc amongst others.

However I now hit new problems with DeployStudio Runtime trying to run a workflow. These basically boil down to the new brutally vicious settings in Security & Privacy and specifically what apps are allowed to modify folders.

I have tried adding Terminal and DeployStudio Runtime and even ditto to the list of apps allowed full disk access but this still does not seem sufficient. I get errors like the following.

2019-10-14 12:11:19.397 DeployStudio Runtime.bin[4102:45231] /usr/bin/ditto --rsrc "/tmp/DSNetworkRepository/Packages/Catalina Touch-1.0.pkg" "/Volumes/Macintosh HD/etc/deploystudio/ds_packages/Catalina Touch-1.0.pkg" 2>&1
2019-10-14 12:11:19.405 DeployStudio Runtime.bin[4102:45231] ditto: /private/tmp/DSNetworkRepository/Packages/./Catalina Touch-1.0.pkg: Operation not permitted

The above is when a DeployStudio workflow tries to copy an installer package configured as a postponed execution. Because it cannot be 'dittod' it then of course cannot be chmod configured because it does not exist in the destination folder.

Similarly if I try running a workflow command to copy a file these also fail.

2019-10-14 10:15:19.547 DeployStudio Runtime.bin[4102:20017] /usr/bin/ditto --rsrc /tmp/DSNetworkRepository/Files/Preferences "/Volumes/Macintosh HD/System/Library/User Template/English.lproj/Library/Preferences" 2>&1
2019-10-14 10:15:19.553 DeployStudio Runtime.bin[4102:20017] ditto: /private/tmp/DSNetworkRepository/Files/Preferences/.: Operation not permitted

I would say most of these issues boil down to ditto being blocked and this causing subsequent errors. I shall have to try some alternatives to ditto such as mounting a disk image from a network share containing the installer pkg or the cat-eof script commands.

Note: All 'postponed' scripts and pkgs require using the ditto command and hence are currently blocked. :(

Last edited by jelockwood (2019-10-14 13:13:36)

Offline

#8 2019-10-14 13:30:59

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

Ok some progress.

If I give /bin/sh full disk access this lets DeployStudio successfully run shell scripts which will cause the creation of files. Based on this a similar process may apply for file copy and install packages but I have to find which process is involved.

Whilst this might sound like it is significantly reducing the security this should only apply to the special Catalina boot drive being used to run DeployStudio Runtime and not the one being built and configures.

Offline

#9 2019-10-14 13:33:48

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

Looking promising, it looks like /bin/sh is the parent process for all the DeployStudio tasks i.e. copy, install, scripts etc.

Offline

#10 2019-10-14 14:02:20

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

Ok, shell scripts and copy commands seem to work now that /bin/sh is given full disk access however postponed installer packages seem busted still.

Very few installer packages can be run as non-postponed due to their structure.

(It would be helpful for others to jump in and contribute.)

Offline

#11 2019-10-16 11:35:25

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

Ok, latest update.

It seems the --volume option for the startosinstall command has also been removed from the Catalina installer. Previously one could boot from driveA run the command and tell it to install on to driveB.

Now without that option this is no longer possible. This sadly is yet another hostile move by Apple. :(

What's more clearly the full Catalina installer _can_ install on to a different drive since the GUI interface allows this.

I am now trying AutoDMG to build an image that can be restored/run by DeployStudio and to add some installers to that image. Going back to AutoDMG is a bit of a step back in time as it is what I used prior to using a script to trigger startosinstall.

Without a further solution it will still be the case that I cannot run a normal DeployStudio workflow to configure settings etc. If AutoDMG works I will have to wrap the basics in my own installer package to use with AutoDMG e.g. an installer to install MunkiTools and before that set all the basic settings like the Munki repo.

Hmm, just thought this still leaves the annoying inability to autoname the Mac based on the DeployStudio database. In fact I have not seen any tools even Jamf to autoname machines based on a database. A script could name based on say serial number but not some other company scheme like an asset number.

Apple are becoming _really_ anti enterprise admin :( They clearly seem to think everyone is a home user using iCloud for everything which is far from true in enterprises. :(

Update: Whilst the createuser pkg works the Jamf QuickAdd pkg does not work with AutoDMG.

Last edited by jelockwood (2019-10-16 11:47:28)

Offline

#12 2019-10-17 16:22:32

jeremyd
Member
Registered: 2017-07-21

Re: DeployStudio, Mojave and Catalina

Were you able to get an AutoDMG image to restore correctly using deploy studio? I couldn't even get my AutoDMG to make 10.15 catalina at all. It errored out right away. I'm curious if it could even restore the dmg file considering Catalina now has two volumes and they are joined with a Volume Group flag.

I'm attempting a process that using a freshly installed OS mac.. and then using deploy studio to just rsync the files from a DMG of the "data" drive on a configured mac to replace the files on the Data Drive of the freshly installed mac.

Example:
hdiutil attach /tmp/DSNetworkRepository/Masters/APFS/FreshCatalina-Data-PreConfigured.i386.apfs.dmg -mountpoint /Volumes/DataMount
rsync -xrlptgoEvHS --delete "/Volumes/DataMount/" "/Volumes/Macintosh HD - Data/"

I'm still messing with that part. I haven't even moved to the postflight steps like the rename and bind to domain steps of DeployStudio.

Offline

#13 2019-10-17 17:32:46

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

@jeremyd

I did not bother trying to restore it but I did find out how to successfully create a Catalina image using AutoDMG.

AutoDMG has been updated to support making a macOS Catalina image but as you (and I) discovered it is not as straight forward as before. The magic trick is that the Install Catalina app used as the source for AutoDMG _must_ be inside a disk image. Therefore you cannot use the copy in /Applications as we are both used to doing.

For me this seemed to solve that part of things.

As it happens I also use Greg Neagle's installinstallmacos.py script to download the macOS installer and this creates it in a disk image anyway.

Offline

#14 2019-10-23 20:11:01

ChristopherOsborn
Member
From: Boulder, CO
Registered: 2018-10-29

Re: DeployStudio, Mojave and Catalina

I am following your Catalina progress with interest.

Just a heads up: I am successfully restoring "fat" APFS Mojave images to my lab (2016 iMacs - obviously non T2 systems). I have it running via netboot right now. The only thing I can't get to work properly is restoring to Fusion APFS formatted drives (but I have some ideas on how to work around this).

Offline

#15 2019-10-23 21:07:33

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

@ChristopherOsborn
I am currently trying to get an installer package to run as part of AutoDMG to create a user account. This installer has been created using Greg Neagle's pycreateuserpkg script. Historically this type of installer has worked fine with AutoDMG.

It seems that Catalina might have far more stringent requirements for installer packages triggered during the installation of the OS. I am waiting for our in-house developer to sign it using our official Apple developer certificate so I can try this again.

If I get this installer to work I will then try some additional installers to replace aspects normally done via a DeployStudio workflow.

I have not tried now or in the past restoring to a Fusion drive but perhaps if you have Fusion drive pair setup and a workflow which does not format or partition it but merely restores the image it might work. DeployStudio's format/partition step does not seem to support APFS so I pre-format using Disk Utility although this has been on single non-Fusion drive setups.

T2 equipped Macs are possible by having a bootable external USB drive running Mojave or Catalina and running the DeployStudio Runtime program on it and then running the restore image workflow. This of course means turning secure boot off at least whilst doing this.

Offline

#16 2019-10-25 10:48:07

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

A summary of AutoDMG issues for Catalina.

1. You need the beta version of AutoDMG
2. The Catalina installer used as the source for AutoDMG _must_ be inside a disk image
3. Assuming you are building an image to restore to "Macintosh HD" you need to ensure that no existing drive called "Macintosh HD" is mounted whilst using AutoDMG
4. If an existing Catalina drive called "Macintosh HD" exists you must also eject the second copy used for the Data in Catalina - remember with Catalina "Macintosh HD" is now in two parts the OS part and the Data part, this second Data part is invisibly mounted so you cannot see it in the Finder and need to use Disk Utility to unmount it

I have been running AutoDMG from a USB drive running Catalina but with a different name so it does not conflict with "Macintosh HD". AutoDMG gets very confused if it sees an existing "Macintosh HD" because it is trying to name the disk image the same and access the disk image via its script and therefore gets confused as which to use.

Last edited by jelockwood (2019-10-25 10:53:22)

Offline

#17 2019-10-25 17:15:24

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

Big progress today!

I am using AutoDMG to build a Catalina image
I am using a tool created by Richard Trougton aka 'Der Flounder' to run multiple installer packages, his tool is a special installer package and is run by AutoDMG, see - https://derflounder.wordpress.com/2013/05/13/first-boot-package-install-pkg/
One of the installer packages I have created myself runs a script which copies some (included) files, sets initial settings and then runs some additional scripts
Other installers include a pkg generated by Greg Neagle's pycreateuserpkg and Greg Neagle's Munki tools installer
I am using a Catalina USB boot drive to run DeployStudio Runtime
I am using DeployStudio to restore the AutoDMG generated image

The above results in a Mac booting in to a fresh Catalina installation with a local Admin account, my basic settings and Munki. It is pretty much at a state from which I would then let it do a DEP enrolment.

I have written my own script to retrieve the Mac computer name from DeployStudio and rename the Mac - this is run by my Installer package mentioned above although having forgotten to include the step to disable the Apple Setup Assistant the results were overwritten by it so I need to retest this.

There is still some other tidying up to do with my settings script and I need to add a step to disable the Apple Setup Assistant but it looks a usable solution for Catalina for me.

Note: As far as I can see AutoDMG does not trigger any firmware updates. You might have to do a normal OS install the first time but assuming you get it booting any subsequent macOS updates e.g. 10.15.1 would include firmware updates which would run as normal.

PS. As a bonus since DeployStudio is not being used to run a full blown workflow it does not incorrectly change the permissions on /Library/Application Support which would upset Sophos Anti-Virus.

Offline

#18 2019-10-31 21:50:15

jeremyd
Member
Registered: 2017-07-21

Re: DeployStudio, Mojave and Catalina

After restoring my Catalina images using an Sh script that runs as the first step of a task, I then have to make sure both the System and the Data drive are fully mounted. I also had issues with the postflight packages and script running, but after making sure both volumes were mounted the deploystudio post flight files copied down and then ran sucessfully on first boot.

Offline

#19 2019-10-31 23:01:36

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

Sounds like you have made progress jeremyd. How did you ensure that both the system and data drives were mounted first. Did you manually mount them or mount them in a script?

Offline

#20 2019-11-01 15:08:25

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

@jeremyd
Yep I am still not clear on how you are doing what you describe.

My finding was that -

a) The boot drive running DeployStudio Runtime has to be running Catalina itself and this means a local USB drive
b) I could have the target Macintosh HD mounted and restore an AutoDMG image to it with no customisation
c) I could not tell DeployStudio Runtime to run a workflow which runs scripts and installers because it could not copy all the items to the target drive due to SIP and Privacy settings

If you can explain how you manage the later that would be a big help.

Offline

#21 2019-11-01 16:59:51

jeremyd
Member
Registered: 2017-07-21

Re: DeployStudio, Mojave and Catalina

I did create a bootable external Hard Drive running Catalina. It is set to autoboot to a user account with all the privacy settings configured to allow terminal/sh/DeployStudio etc full disk access. I then run deploystudio as a root user (using sudo) just to make sure it has full rights to everything.

I guess you could potentially do this with another mac connected via target disk mode as well. So this is how I get a "imaging environment" up and running.

To capture the first image of a machine I want to replicate I run this script via deploy studio (or manually works too).
-----Capture Catalina Image---------------------------
#!/bin/sh
#Create SparseBundle to use to copy Image To
hdiutil create /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle -volname "MacPrep" -fs apfs -size 50GB
hdiutil mount /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle
#Figure out the disk number of the mounted temp image
disknumber=$(diskutil list MacPrep | grep Scheme | awk {'print $8'})
diskutil apfs deletevolume MacPrep
echo $disknumber
asr --source /dev/disk1s1 --target /dev/$disknumber
#unmount restored disk in order to convert it
diskutil unmountdisk $disknumber
#Pause 5 seconds while waiting for Unmount
sleep 5
#Eject the Image File
hdiutil detach $disknumber
sleep 5
#convert from sparseBundle to DMG
hdiutil convert /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle -format UDZO -o /tmp/DSNetworkRepository/Masters/APFS/TempImage-Converted.DMG
exit 0
-----------------------
If the above run successfully should now have a working image of your original stored in the DeployStudio Repository. You can then verify all is there and delete the .sparsebundle and then rename the TempImage to your name of choice.

Now in order to restore the disk image to another mac I boot the "imaging environment" on it and run a deploystudio sequence with the following script:

------RESTORE Catalina Image------
#!/bin/sh
#Delete and Re-partition creating a APFS Container and temp volume, then delete Temp Volume
diskutil partitionDisk /dev/disk0 1 gpt apfs "MacOSPrep" 100%
diskutil apfs deletevolume MacOSPrep
# Must be running on a 10.15 boot disk for next step to work and restore both System and Data Drives
sudo asr --source /tmp/DSNetworkRepository/Masters/APFS/CatalinaImage-Converted.dmg --target /dev/disk1 -noverify
#make sure Disk is Mounted to configure the rest
diskutil mountDisk disk1
#Pause 2 seconds to allow disk to mount
sleep 2
exit 0
---------------------------------------
After that script runs you shoudl have a restored image with both System and Data drive mounted. You can then setup your task to complete any other steps required such as do the rename host bind domain and install any pkgs.

Hope this helps!

Jeremy

Last edited by jeremyd (2019-11-01 17:02:06)

Offline

#22 2019-11-01 18:26:43

jelockwood
Member
Registered: 2009-11-11

Re: DeployStudio, Mojave and Catalina

@jeremyd
Thanks for the reply. I am using AutoDMG to make a fresh OS image. It was the last step of then running a 'normal' DeployStudio workflow that I was struggling with.

I had a further thought and I suspect what is happening is that after restoring the AutoDMG image it has not yet mounted the equivalent Data volume. I will look in to and check that when I get back from a brief holiday. Even if I give up on that I now have my alternative approach of using AutoDMG with the Richard Troughton's 'first-boot' installer package completely working to do what I want.

Offline

#23 Today 00:35:02

chrisb42
Member
Registered: 2014-03-14

Re: DeployStudio, Mojave and Catalina

Wow, you're going through a lot of work to keep using DeployStudio.

For Catalina and machines that don't boot High Sierra anymore, I've developed a process using MDS – Mac Deploy Stick (http://twocanoes.com/products/mac/mac-deploy-stick/). I found that it makes life quite a bit easier by letting you create users in a dialog that it then generates as a package, using the regular installers, and also allowing various scripts to run. I could actually re-use quite a bit of my scripts used with DS to approximate a setup that DS would produce. Of course, it also helped that I have used Munki with DS already for a year, so that takes care of initial installs and updating apps in general.

Offline

Board footer

Powered by FluxBB