Stuff that has not yet gone into the official build.
Post a reply

refractasnapshot re-run on a previously saved filesystem

Sat Apr 06, 2013 12:05 am

I need ability to rebuild a live image from a saved filesystem, such as refractasnapshot creates.

For example my current Wheezy system. When it was fairly new I made a snapshot. Now I want the live-image updated and tailored specifically for mobile use (not in the same way as the current running system) This involves some manual additions/deletions and chroot for apt stuff (including dist-upgrade)

Previously this was done manually or with custom scripts. However refracta snapshot itself can have a "nocopy" option with some quite minor code additions. Posted is a diff against 9.0.8-2 showing what I did.

Code:
179a180,185
>
> else
>    if [ "$nocopy" = "yes" ]; then
>    $DIALOG --$ERROR --title="Refracta Snapshot" --text=" Error.. No saved filesystem copy was found.. "
>    exit 1
>    fi
190a197
>
196a204,220
>
> # Don't use /media/*  for $snapshot_dir unless it is a mounted filesystem
> snapdir_is_remote=$(echo ${snapshot_dir}| awk -F / '{print "/" $2 "/" $3}' |grep /media/)
>
> if [ -n "$snapdir_is_remote" ] && cat /proc/mounts|grep -q ${snapdir_is_remote}; then
>    echo "$snapdir is mounted"
> else
>    $DIALOG --$ERROR --title="Refracta Snapshot" --text=" Error.. The selected snapshot work directory cannot be accessed.. "
>    exit 1
> fi
>
>
> if [ "$nocopy" = "yes" ]; then
>    return
> fi
>
>
278c302
<    $DIALOG --$INFO --text="Excludes file was set to default values.
---
>    $DIALOG --$INFO --title="Refracta Snapshot" --text="Excludes file was set to default values.
281c305
<    $DIALOG --$WARNING --text="Something went wrong. You should check your excludes file.
---
>    $DIALOG --$WARNING --title="Refracta Snapshot" --text="Something went wrong. You should check your excludes file.
296a321
>   FALSE 04 "Use a previously saved filesystem copy." \
309a335,342
>
> if $(echo $opts | grep -q 04); then
>    nocopy="yes"
>    save_work="yes"
>    kernel_image="${work_dir}/myfs/vmlinuz"
>    initrd_image="${work_dir}/myfs/initrd.img"
> fi
>
335c368
< $DIALOG --$INFO --text="This may take a moment while the program checks for free space.    "
---
> $DIALOG --$INFO --title="Refracta Snapshot" --text="This may take a moment while the program checks for free space.    "
347a381,383
>
> copy_filesystem () {
>
352a389,401
> # Copy the filesystem
> rsync -av / myfs/ --delete --exclude="$work_dir" \
>  --exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
>  | tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
> kill $(pgrep $DIALOG)
>
> }
>
>
> if ! [ "$nocopy" = "yes" ]; then
> copy_filesystem
> fi
>
361,367d409
< # Copy the filesystem
< rsync -av / myfs/ --delete --exclude="$work_dir" \
<  --exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
<  | tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
< kill $(pgrep $DIALOG)
<
<
428c470
<     $DIALOG --$QUESTION --title=Edit Boot Menu --${BUTTON0}="Yes"${BUTTON0NUM} --${BUTTON1}="No"${BUTTON1NUM} \
---
>     $DIALOG --$QUESTION --title="Edit Boot Menu" --${BUTTON0}="Yes"${BUTTON0NUM} --${BUTTON1}="No"${BUTTON1NUM} \


Another advantage: The original snapshot was 3.5 GB. After dealing manually with some excess stuff which is probably beyond the reasonable scope of snapshot's rsync-exclude list I got it down to 1.1 GB (amazing what lurks in those hidden "dot" files)

One word of warning to anyone who tries this.. Be careful what and how you delete, there might be symlinked directories to outside the saved FS in (most likely but could be elsewhere) $FS/home.. "rm -rf" horror stories are well documented elsewhere.

Note also my patch is "experimental" (tested here and one time only with zero guarantee against data loss) however the idea at least might be a useful addition to official Refractasnapshot. It includes a few other things like fixing some dialog windows that didn't display well and a check for, if a custom "work" dir on another filestem is used, it is actually plugged and mounted.

Re: refractasnapshot re-run on a previously saved filesystem

Sat Apr 06, 2013 1:02 pm

Code:
>    kernel_image="${work_dir}/myfs/vmlinuz"
>    initrd_image="${work_dir}/myfs/initrd.img"

I think this should cause errors with the cp command trying to copy a file to itself. Are you seeing that, and if so, is it a problem?

Actually, I think I'll just go ahead and make copy_kernel a function like you did with copy_filesystem, and they both can run (or not) together. Then kernel_image and initrd_image don't need to be redefined.
Code:
# Let iso/, vmlinuz and initrd.img get copied, even if work_dir was saved,
# in case they have changed, unless $nocopy = yes.
copy_kernel () {
rsync -a "$iso_dir"/ "$work_dir"/iso/
cp "$kernel_image" "$work_dir"/iso/live/
cp "$initrd_image" "$work_dir"/iso/live/
}

# Copy the filesystem
copy_filesystem () {

rsync -av / myfs/ --delete --exclude="$work_dir" \
--exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
| tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
kill $(pgrep $DIALOG)
}

if ! [ "$nocopy" = "yes" ]; then
   copy_kernel
   copy_filesystem
fi

Re: refractasnapshot re-run on a previously saved filesystem

Sun Apr 07, 2013 1:09 am

I think this should cause errors with the cp command trying to copy a file to itself. Are you seeing that, and if so, is it a problem?

Not sure what that means, it's getting copied from $work_dir/myfs/ to $work_dir/iso/live/ .. if it's a link a straight cp uses the link's target and overwrites one with the same name in the destination. No such errors in the log, their date stamps show they were copied at that time and the iso boots

But it's wrong anyway because $work_dir/myfs/initrd.img is absolute link to /boot/whatever.img (= the running system's /boot .. in this scenario we want the one from /boot of the saved fs). Curiously vmlinuz link is relative (boot/whatever). In fact this running system has the same kernel but could have been different.Thanks for the prompt to review that.

Something like this might be better, will try later:

Code:
# nocopy mode
kernel_image="$(ls ${work_dir}/myfs/boot|grep -m 1 vmlinuz)"
initrd_image="$(ls ${work_dir}/myfs/boot|grep -m 1 initrd)"

# later
cp ${work_dir}/myfs/boot/${kernel_image} ${work_dir}/iso/live/vmlinuz
cp ${work_dir}/myfs/boot/${initrd_image} ${work_dir}/iso/live/initrd.img

Later to work out how to deal with a second kernel, might need to boot something that can't handle PAE

Experimental only so far!

Re: refractasnapshot re-run on a previously saved filesystem

Sun Apr 07, 2013 1:34 pm

Oh. It means that I wasn't looking at the code. Sorry.

Would it help to split up $kernel_image and $kernel_path, so that $kernel_image is the file name (vmlinuz for a default) and the script could adjust $kernel_path for nocopy?

These lines should also be dealt with for nocopy:
Code:
# @@@@  Warning: This will replace these files in custom iso_dir  @@@@@
#copy some isolinux stuff from the system to the snapshot
rsync -a /usr/lib/syslinux/isolinux.bin "$iso_dir"/isolinux/
rsync -a /usr/lib/syslinux/vesamenu.c32 "$iso_dir"/isolinux/


Is rsync really needed at all for nocopy? Would it make sense to skip that and go right to mksquashfs?

Re: refractasnapshot re-run on a previously saved filesystem

Sun Apr 07, 2013 3:07 pm

And should the nocopy option make the script pause right before mksquashfs (same as edit_boot_menu=yes) for manual tweaks?

Re: refractasnapshot re-run on a previously saved filesystem

Mon Apr 08, 2013 1:05 am

Do you want the package list to be based on the previously saved copy? Right now, 'dpkg -l' is run on the host system. I think it could chroot $work_dir for the nocopy option.

change this:
Code:
# Prepend the dir name with a constant,
# so you can find and delete the old ones
# that might have different snapshot basenames.
dir_prefix="pkglist"

rm -r "$work_dir"/iso/"$dir_prefix"*
mkdir -p "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"
dpkg -l | grep "ii" | awk '{ print $2 }' > "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"/package_list


to this?
Code:
# Prepend the dir name with a constant,
# so you can find and delete the old ones
# that might have different snapshot basenames.
dir_prefix="pkglist"

rm -r "$work_dir"/iso/"$dir_prefix"*
mkdir -p "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"
if [[ $nocopy = "yes" ]] ; then
    chroot "$work_dir" dpkg -l | grep "ii" | awk '{ print $2 }' > "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"/package_list
else
    dpkg -l | grep "ii" | awk '{ print $2 }' > "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"
fi
   

Re: refractasnapshot re-run on a previously saved filesystem

Mon Apr 08, 2013 1:13 am

Here's the rsync section. I think it's all set - no rsync will run if $nocopy = yes. There are a few other preparations that are done after that, for pmount, updatedb, clamav, root ssh, snapshot name, and then it gets squashed.
Code:
# The real work starts here
cd "$work_dir"

# @@@@  Warning: This will replace these files in custom iso_dir  @@@@@
#copy some isolinux stuff from the system to the snapshot
copy_isolinux () {
rsync -a /usr/lib/syslinux/isolinux.bin "$iso_dir"/isolinux/
rsync -a /usr/lib/syslinux/vesamenu.c32 "$iso_dir"/isolinux/
}

# Let iso/, vmlinuz and initrd.img get copied, even if work_dir was saved,
# in case they have changed, unless $nocopy = yes.
copy_kernel () {
rsync -a "$iso_dir"/ "$work_dir"/iso/
cp "$kernel_image" "$work_dir"/iso/live/
cp "$initrd_image" "$work_dir"/iso/live/
}

# Copy the filesystem
copy_filesystem () {

rsync -av / myfs/ --delete-excluded --exclude="$work_dir" \
--exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
| tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
kill $(pgrep $DIALOG)
}

if ! [ "$nocopy" = "yes" ]; then
   copy_isolinux
   copy_kernel
   copy_filesystem
fi


Another thought was to force a pause before squashing, but I didn't do this one:
Code:
if [[ $nocopy = "yes" ]] ; then
    edit_boot_menu="yes"
fi

Re: refractasnapshot re-run on a previously saved filesystem

Mon Apr 08, 2013 2:41 pm

Don't see a reason to pause, the whole idea is ability do do some manual tweaks in the saved fs before rebuild, presumable that is already done (including menus if needed) It shouldn't matter if those few preperations get run again

The package lists would be different only if a user had chrooted it with bind mounts and run apt (rather than simple file additions/deletions).

Doing that is possible but gets more complex. The bind mounts can be difficult to umount (but a reboot sorts them) because services might have been started in the chroot. Also some stuff listed in the rsync excludes gets written (log files, apt files, bash history..) and would need finding and deleting manually.

If you know how to handle that you don't need help updating a package list!

Re: refractasnapshot re-run on a previously saved filesystem

Tue Apr 09, 2013 1:58 pm

refractasnapshot-9.0.9 (base and gui packages)
http://distro.ibiblio.org/refracta/testing/

Code:
  * Added test for remote, unmounted snapshot_dir or work_dir
  * Added more items to excludes, especially root's hidden files.
  * Changed rsync option '--delete' to '--delete-excludes'
  * Added nocopy option to bypass rsync.


To use the nocopy option with the text-only version, run it with the -n (--nocopy) option on the command-line.
Code:
refractasnapshot -n

Re: refractasnapshot re-run on a previously saved filesystem

Tue Sep 03, 2013 9:00 am

the gui version begs inclusion of some explanatory text.
When it pauses, a yad(?) screen presents 5 checkboxes with 5 too-short "labels".
User wonders "Can I checkmark more than one?" and (later) "why am I seeing this same screen again?"
Also, it would be helpful if some text (technical, or non-technical, in addition to progress bar) displayed what step/action the script is currently performing.
Post a reply