368 lines
16 KiB
Diff
368 lines
16 KiB
Diff
From cb1bce51a4fc8a6e194673c8e7dd826541561f0a Mon Sep 17 00:00:00 2001
|
|
From: Daniel McKnight <daniel@mcknight.tech>
|
|
Date: Thu, 28 Mar 2024 05:08:39 +0000
|
|
Subject: [PATCH 1/2] Write intro and outline instructions
|
|
|
|
---
|
|
2024-03-27_Nextcloud-S3-Migration.md | 60 ++++++++++++++++++++++++++++
|
|
1 file changed, 60 insertions(+)
|
|
create mode 100644 2024-03-27_Nextcloud-S3-Migration.md
|
|
|
|
diff --git a/2024-03-27_Nextcloud-S3-Migration.md b/2024-03-27_Nextcloud-S3-Migration.md
|
|
new file mode 100644
|
|
index 0000000..d600b6c
|
|
--- /dev/null
|
|
+++ b/2024-03-27_Nextcloud-S3-Migration.md
|
|
@@ -0,0 +1,60 @@
|
|
+---
|
|
+date: 2024-03-27
|
|
+title: Migrating Nextcloud from Local Storage to S3
|
|
+tags:
|
|
+ - homelab
|
|
+ - s3
|
|
+ - Nextcloud
|
|
+---
|
|
+
|
|
+As I continue to investigate [migration from Unraid to XCP-ng](https://blog.mcknight.tech/2024/01/30/Compute_Setup/)
|
|
+(or maybe Proxmox), I am taking a closer look at what storage resources my containers are still using on my Unraid
|
|
+system. Unsurprisingly, Nextcloud is using a significant amount of persistent storage and I found that moving that
|
|
+data to a remote share is difficult due to Nextcloud's file permission requirements. Looking for a solution, I found
|
|
+that Nextcloud supports S3 storage and, while not officially supported, there are ways to move from local storage to
|
|
+S3 while preserving configuration. I will admit that much of this was just copy/pasted from other guides which I'll
|
|
+link below, but there were a few gotchas related to doing this with containers that I'll highlight.
|
|
+
|
|
+## Why to migrate storage
|
|
+As I mentioned, my primary reason for migrating from local to S3 storage is to simplify running Nextcloud in a
|
|
+container with primary storage residing on a different system. This is also a fun project for me to learn a
|
|
+little more about how S3 works (this is my first application using S3 storage 🎉). I'm using MinIO on TrueNAS
|
|
+which was practically a one-click deployment; I might have a more detailed post about this in the future.
|
|
+
|
|
+## Preparing for migration
|
|
+Fist things first, configure a bucket for Nextcloud (and only Nextcloud) to use; I've named mine "nextcloud".
|
|
+I only have a few users on my Nextcloud instance and I'm the only one who uses it regularly, so I was able to
|
|
+prevent any changes during the migration by closing any desktop applications that sync and not taking any
|
|
+photos on my phone. If you have more users or activity, I would recommend putting Nextcloud into maintenance
|
|
+mode before migrating things (`occ maintenance:mode --on`).
|
|
+
|
|
+## Migrating from local to S3
|
|
+Other guides I found made some assumptions about Nextcloud and SQL running in the same environment, which
|
|
+isn't true in my case where they're in containers. They also assumed the `aws` cli could be used where
|
|
+Nextcloud is running which is also not true in my case. The code snippets below have been modified from
|
|
+other examples to fit my use case.
|
|
+
|
|
+### Get files from SQL
|
|
+- export meta files
|
|
+- export user files
|
|
+- ensure valid paths
|
|
+
|
|
+### Move files to upload directory
|
|
+- filenames
|
|
+- NFS Mount (No SMB)
|
|
+- Upload to S3
|
|
+
|
|
+### Nextcloud config updates
|
|
+...
|
|
+
|
|
+### SQL database updates
|
|
+...
|
|
+
|
|
+
|
|
+## Validating Changes
|
|
+- user login/errors
|
|
+- file availability
|
|
+
|
|
+## References
|
|
+- [mrAceT migration scripts](https://github.com/mrAceT/nextcloud-S3-local-S3-migration)
|
|
+- [Nextcloud issue with guide](https://github.com/nextcloud/server/issues/25781)
|
|
\ No newline at end of file
|
|
--
|
|
GitLab
|
|
|
|
|
|
From 055524280f40457eacf5d015dcdbf00f54966fab Mon Sep 17 00:00:00 2001
|
|
From: Daniel McKnight <daniel@mcknight.tech>
|
|
Date: Fri, 29 Mar 2024 05:40:07 +0000
|
|
Subject: [PATCH 2/2] Finish Nextcloud-S3 Post
|
|
|
|
---
|
|
2024-03-27_Nextcloud-S3-Migration.md | 60 ------
|
|
...s.md => 2024-03-27_Shell-Customizations.md | 0
|
|
2024-03-28_Nextcloud-S3-Migration.md | 192 ++++++++++++++++++
|
|
3 files changed, 192 insertions(+), 60 deletions(-)
|
|
delete mode 100644 2024-03-27_Nextcloud-S3-Migration.md
|
|
rename 2024-03-10_Shell-Customizations.md => 2024-03-27_Shell-Customizations.md (100%)
|
|
create mode 100644 2024-03-28_Nextcloud-S3-Migration.md
|
|
|
|
diff --git a/2024-03-27_Nextcloud-S3-Migration.md b/2024-03-27_Nextcloud-S3-Migration.md
|
|
deleted file mode 100644
|
|
index d600b6c..0000000
|
|
--- a/2024-03-27_Nextcloud-S3-Migration.md
|
|
+++ /dev/null
|
|
@@ -1,60 +0,0 @@
|
|
----
|
|
-date: 2024-03-27
|
|
-title: Migrating Nextcloud from Local Storage to S3
|
|
-tags:
|
|
- - homelab
|
|
- - s3
|
|
- - Nextcloud
|
|
----
|
|
-
|
|
-As I continue to investigate [migration from Unraid to XCP-ng](https://blog.mcknight.tech/2024/01/30/Compute_Setup/)
|
|
-(or maybe Proxmox), I am taking a closer look at what storage resources my containers are still using on my Unraid
|
|
-system. Unsurprisingly, Nextcloud is using a significant amount of persistent storage and I found that moving that
|
|
-data to a remote share is difficult due to Nextcloud's file permission requirements. Looking for a solution, I found
|
|
-that Nextcloud supports S3 storage and, while not officially supported, there are ways to move from local storage to
|
|
-S3 while preserving configuration. I will admit that much of this was just copy/pasted from other guides which I'll
|
|
-link below, but there were a few gotchas related to doing this with containers that I'll highlight.
|
|
-
|
|
-## Why to migrate storage
|
|
-As I mentioned, my primary reason for migrating from local to S3 storage is to simplify running Nextcloud in a
|
|
-container with primary storage residing on a different system. This is also a fun project for me to learn a
|
|
-little more about how S3 works (this is my first application using S3 storage 🎉). I'm using MinIO on TrueNAS
|
|
-which was practically a one-click deployment; I might have a more detailed post about this in the future.
|
|
-
|
|
-## Preparing for migration
|
|
-Fist things first, configure a bucket for Nextcloud (and only Nextcloud) to use; I've named mine "nextcloud".
|
|
-I only have a few users on my Nextcloud instance and I'm the only one who uses it regularly, so I was able to
|
|
-prevent any changes during the migration by closing any desktop applications that sync and not taking any
|
|
-photos on my phone. If you have more users or activity, I would recommend putting Nextcloud into maintenance
|
|
-mode before migrating things (`occ maintenance:mode --on`).
|
|
-
|
|
-## Migrating from local to S3
|
|
-Other guides I found made some assumptions about Nextcloud and SQL running in the same environment, which
|
|
-isn't true in my case where they're in containers. They also assumed the `aws` cli could be used where
|
|
-Nextcloud is running which is also not true in my case. The code snippets below have been modified from
|
|
-other examples to fit my use case.
|
|
-
|
|
-### Get files from SQL
|
|
-- export meta files
|
|
-- export user files
|
|
-- ensure valid paths
|
|
-
|
|
-### Move files to upload directory
|
|
-- filenames
|
|
-- NFS Mount (No SMB)
|
|
-- Upload to S3
|
|
-
|
|
-### Nextcloud config updates
|
|
-...
|
|
-
|
|
-### SQL database updates
|
|
-...
|
|
-
|
|
-
|
|
-## Validating Changes
|
|
-- user login/errors
|
|
-- file availability
|
|
-
|
|
-## References
|
|
-- [mrAceT migration scripts](https://github.com/mrAceT/nextcloud-S3-local-S3-migration)
|
|
-- [Nextcloud issue with guide](https://github.com/nextcloud/server/issues/25781)
|
|
\ No newline at end of file
|
|
diff --git a/2024-03-10_Shell-Customizations.md b/2024-03-27_Shell-Customizations.md
|
|
similarity index 100%
|
|
rename from 2024-03-10_Shell-Customizations.md
|
|
rename to 2024-03-27_Shell-Customizations.md
|
|
diff --git a/2024-03-28_Nextcloud-S3-Migration.md b/2024-03-28_Nextcloud-S3-Migration.md
|
|
new file mode 100644
|
|
index 0000000..9d9bf31
|
|
--- /dev/null
|
|
+++ b/2024-03-28_Nextcloud-S3-Migration.md
|
|
@@ -0,0 +1,192 @@
|
|
+---
|
|
+date: 2024-03-28
|
|
+title: Migrating Nextcloud from Local Storage to S3
|
|
+tags:
|
|
+ - homelab
|
|
+ - s3
|
|
+ - Nextcloud
|
|
+---
|
|
+
|
|
+As I continue to investigate [migration from Unraid to XCP-ng](https://blog.mcknight.tech/2024/01/30/Compute_Setup/)
|
|
+(or maybe Proxmox), I am taking a closer look at what storage resources my containers are still using on my Unraid
|
|
+system. Unsurprisingly, Nextcloud is using a significant amount of persistent storage and I found that moving that
|
|
+data to a remote share is difficult due to Nextcloud's file permission requirements. Looking for a solution, I found
|
|
+that Nextcloud supports S3 storage and, while not officially supported, there are ways to move from local storage to
|
|
+S3 without having to start over from scratch. I will admit that much of this was heacily inspired by other guides
|
|
+which I've link below, but there were a few gotchas related to doing this with containers in Unraid that I'll highlight.
|
|
+
|
|
+## Why to migrate storage
|
|
+As I mentioned, my primary reason for migrating from local to S3 storage is to simplify running Nextcloud in a
|
|
+container with primary storage residing on a different system. This is also a fun project for me to learn a
|
|
+little more about how S3 works (this is my first application using S3 storage 🎉). I'm using MinIO on TrueNAS
|
|
+which was practically a one-click deployment; I followed
|
|
+[this video from Lawrence Systems](https://youtu.be/uIm41PhGEgQ?si=0X1FZgsNfZisnaKw).
|
|
+
|
|
+## Preparing for migration
|
|
+Fist things first, configure a bucket for Nextcloud (and only Nextcloud) to use; I've named mine "nextcloud".
|
|
+I only have a few users on my Nextcloud instance and I'm the only one who uses it regularly, so I was able to
|
|
+prevent any changes during the migration by closing any desktop applications that sync and not taking any
|
|
+photos on my phone. If you have more users or activity, I would recommend putting Nextcloud into maintenance
|
|
+mode before migrating things (`occ maintenance:mode --on`).
|
|
+
|
|
+## Migrating from local to S3
|
|
+Other guides I found made some assumptions about Nextcloud and SQL running in the same environment, which
|
|
+isn't true in my case where they're in containers. They also assumed the `aws` cli could be used where
|
|
+Nextcloud is running which is also not true in my case since Unraid doesn't have that package available.
|
|
+The code snippets below have been modified from other examples to fit my use case.
|
|
+
|
|
+### Get files from SQL
|
|
+My Nextcloud setup uses MariaDb, so the first step is to export the list of files, along with some identifiers
|
|
+Nextcloud uses, to be uploaded. My database is called `nextcloud` and the `/config` directory is mounted to the
|
|
+host. `/mnt/user/Nextcloud` on the host system is mounted to `/data` in the Nextcloud container.
|
|
+
|
|
+```shell
|
|
+mysql -p -B --disable-column-names -D nextcloud << EOF > /config/meta_file_list
|
|
+ select concat('urn:oid:', fileid, ' ', substring(id from 8), path)
|
|
+ from oc_filecache
|
|
+ join oc_storages
|
|
+ on storage = numeric_id
|
|
+ where id like 'local::%'
|
|
+ order by id;
|
|
+EOF
|
|
+
|
|
+mysql -p -B --disable-column-names -D nextcloud << EOF > /config/user_file_list
|
|
+ select concat('urn:oid:', fileid, ' ', '/mnt/user/Nextcloud/',
|
|
+ substring(id from 7), '/', path)
|
|
+ from oc_filecache
|
|
+ join oc_storages
|
|
+ on storage = numeric_id
|
|
+ where id like 'home::%'
|
|
+ order by id;
|
|
+EOF
|
|
+
|
|
+sed -i -e "s| /data/| /mnt/user/Nextcloud/|g" meta_file_list
|
|
+```
|
|
+
|
|
+Note that `user_file_list` paths are updated in the `mysql` command and the `meta_file_list` is updated
|
|
+by the `sed` command to be accessible on the host.
|
|
+
|
|
+### Move files to upload directory
|
|
+Since I will be mounting the upload directory to a virtual machine, I opted to copy files rather than
|
|
+create symlinks like the example I used as reference. It doesn't particularly matter where on the host
|
|
+system the files to upload are copied to, but I used `/mnt/user/Nextcloud` which was my existing share
|
|
+for Nextcloud data. I copied `user_file_list` and `meta_file_list` from the previous step to that path.
|
|
+
|
|
+```shell
|
|
+cd /mnt/user/Nextcloud
|
|
+mkdir s3_files
|
|
+cd s3_files
|
|
+
|
|
+while read target source ; do
|
|
+ if [ -f "$source" ] ;
|
|
+then
|
|
+ cp "$source" "$target"
|
|
+ fi
|
|
+done < ../user_file_list
|
|
+
|
|
+ while read target source ; do
|
|
+ if [ -f "$source" ] ;
|
|
+ then
|
|
+ cp "$source" "$target"
|
|
+ fi
|
|
+done < ../meta_file_list
|
|
+```
|
|
+
|
|
+At this point, all of the files are named and placed in the `s3_files` directory, ready to be uploaded
|
|
+to S3. Note that since the filenames contain `:`, this directory will NOT be accessible via an SMB share.
|
|
+I exported it as an NFS share, but if I could install `awscli` on the host system then it wouldn't have
|
|
+been a concern.
|
|
+
|
|
+On my VM where I actually uploaded everything to a bucket named `nextcloud`:
|
|
+
|
|
+```shell
|
|
+sudo mount <SERVER_ADDR>:/mnt/user/Nextcloud /home/$USER/Nextcloud
|
|
+cd ~/Nextcloud
|
|
+aws --endpoint-url <S3_URL> s3 sync s3_files s3://nextcloud
|
|
+```
|
|
+
|
|
+### Nextcloud config updates
|
|
+With all of the files uploaded to S3, Nextcloud needs to be updated to point at the new data source. So,
|
|
+time to put Nextcloud into maintenance mode and start breaking things 🙂. From here on out, we are making
|
|
+potentially destructive changes, so make backups, have a recovery plan, and don't continue
|
|
+until reading to the end first.
|
|
+
|
|
+In the Docker container:
|
|
+
|
|
+```shell
|
|
+occ maintenance:mode --on
|
|
+nano /config/www/nextcloud/config/config.php
|
|
+```
|
|
+
|
|
+Add the following to the Nextcloud config (`/config/www/nextcloud/config/config.php`):
|
|
+
|
|
+```php
|
|
+ 'objectstore' =>
|
|
+ array (
|
|
+ 'class' => 'OC\\Files\\ObjectStore\\S3',
|
|
+ 'arguments' =>
|
|
+ array (
|
|
+ 'bucket' => 'nextcloud',
|
|
+ 'autocreate' => true,
|
|
+ 'key' => '<MY_AWS_KEY>',
|
|
+ 'secret' => '<MY_AWS_SECRET>',
|
|
+ 'hostname' => '<S3_ADDRESS>',
|
|
+ 'port' => 443,
|
|
+ 'use_ssl' => true,
|
|
+ 'region' => 'home',
|
|
+ 'use_path_style' => true,
|
|
+ ),
|
|
+ ),
|
|
+```
|
|
+> Make sure this is before the closing `);` at the end of the file
|
|
+
|
|
+`use_path_style` was required for MinIO, but I read that this may be false for other providers.
|
|
+
|
|
+### SQL database updates
|
|
+The SQL database at this point still contains local file references, so those will need to be
|
|
+updated to the uploaded resources. In the SQL container:
|
|
+```shell
|
|
+mariadb -p -D nextcloud -e "update oc_storages set id = concat('object::user:', substring(id from 7)) where id like 'home::%';"
|
|
+mariadb -p -D nextcloud -e "update oc_storages set id = 'object::store:amazon::nextcloud' where id like 'local::%';"
|
|
+mariadb -p -D nextcloud -e "update oc_mounts set mount_provider_class = 'OC\\\Files\\\Mount\\\ObjectHomeMountProvider' where mount_provider_class like '%LocalHomeMountProvider%';"
|
|
+```
|
|
+
|
|
+### Validating Changes
|
|
+With configuration and database changes completed, its time to take Nextcloud out of maintenance mode
|
|
+and validate changes. In the Nextcloud container:
|
|
+```shell
|
|
+occ maintenance:mode --off
|
|
+```
|
|
+
|
|
+> I also restarted the Nextcloud container here for good measure, but it may not be necessary.
|
|
+
|
|
+Load up Nextcloud in a browser and make sure the page loads, user login works, and most importantly
|
|
+user files are accessible. If this all works, then the migration is complete!
|
|
+
|
|
+## Cleanup local files
|
|
+With migration completed, I chose to remove the `data` container mount since the directory only needs
|
|
+the `.ocdata` file and should only contain logs. On the host system:
|
|
+```shell
|
|
+cp /mnt/user/Nextcloud/.oc_data /mnt/user/appdata/nextcloud/data/
|
|
+chmod 770 /mnt/user/appdata/nextcloud/data/
|
|
+chown nobody:users /mnt/user/appdata/nextcloud/data/
|
|
+```
|
|
+> Note that the ownership is specific to Unraid, but Nextcloud requires permissions `770`.
|
|
+
|
|
+Then, modify Nextcloud's `config.php`:
|
|
+```php
|
|
+'datadirectory' => '/config/data',
|
|
+```
|
|
+
|
|
+Then the `/data` mount can be removed from the Docker container and the container restarted. At this point,
|
|
+the old data share can be removed completely (I'll archive mine as a backup).
|
|
+
|
|
+## Wrap up
|
|
+I have been using my Nextcloud with S3 storage for a few days now and haven't noticed any difference compared
|
|
+to local storage. I like that the storage is accessible via HTTP so I don't have to worry about
|
|
+managing permissions and access from the Nextcloud container to my storage server.
|
|
+
|
|
+## References
|
|
+- [mrAceT migration scripts](https://github.com/mrAceT/nextcloud-S3-local-S3-migration)
|
|
+- [Nextcloud issue with guide](https://github.com/nextcloud/server/issues/25781)
|
|
+- [Nextcloud forum thread](https://help.nextcloud.com/t/migrate-from-local-storage-to-s3/138042)
|
|
\ No newline at end of file
|
|
--
|
|
GitLab
|
|
|