How to transfer all email from mail.ru to your debian 8 + iredmail mail server using imapsync?

Question

Email

Good time of the day!!!
Task: Switch from biz.mail.ru (mail.ru using corporate domain) to your own debian 8 + iredmail (LDAP) server. Number of mailboxes 105.
Done: Installed, configured and tested iRedMail on Debian 8. Everything works fine: A, MX, TXT (DKIM) records are correct, SSL is configured, in short everything works and letters go.
All that's left is "PAILLESS", that is, without disturbing users, transfer all mailboxes in a working rhythm. As a result, the imapsync tool was found. Quite
an interesting and simple utility with the ability to transfer several mailboxes at once.
It is easy to use (in my opinion), since it is written in Perl, except for libraries, nothing else needs to be installed, it works as a "one-liner", but you can enter all the necessary parameters into a script and run it. And you can also configure it to synchronize taking into account the already existing duplicates, that is, it transfers only newly appeared letters ( but there is a BUT ), a good FAQ, in general, a worthwhile thing.
After studying the manual, I launched 4 mailboxes for synchronization (an average of 1.5k letters in the inbox, and a certain number in the rest of the folders). I was pleasantly surprised by the process:
0. There is a connection
1. I counted the available folders on the first email box (from where we transfer)
2. I counted the available folders on the second email box (where we transfer)
3. Compared them and issued a discrepancy, then created the missing folders on the second email box, for the correct transfer of mail ( but there is a second BUT )
4. Started the transfer, the transfer occurs in stages, first one folder - reading the headers, transferring the "body" itself, etc. .d.
5. Displays the final output on the screen - what, where, why and why.
Everything happens beautifully, relatively quickly, BUT
Now BUT :
1. When comparing folders on mailboxes, and as we know, where we transfer this mail.ru from and where we transfer it , iredmail displays the following:

++++ Listing folders
All foldernames are presented between brackets like [X] where X is the foldername.
When a foldername contains non-ASCII characters it is presented in the form
[X] = [Y] where
X is the imap foldername you have to use in command line options and
Y is the uft8 output just printed for convenience, to recognize it.

Host1 folders list:
[&BB4EQgQ,BEAEMAQyBDsENQQ9BD0ESwQ1-] = [Отправленные]
[&BBoEPgRABDcEOAQ9BDA-] = [Корзина]
[&BCEEPwQwBDw-] = [Спам]
[&BCcENQRABD0EPgQyBDgEOgQ4-] = [Черновики]
[INBOX]

Host2 folders list:
[Drafts]
[INBOX]
[Junk]
[Sent]
[Trash]

Folders in host2 not in host1:
[Trash]
[Sent]
[Junk]
[Drafts]

This suggests that when matching folders, only INBOX matched, the rest will be created, this does not suit us, since we will have to manually transfer from the created folders to existing ones, it turned out like this:

In the manual, I found the --regextrans2 option to manually match folder names. No matter how hard I try, nothing happens, folders are still created and the corresponding letters are transferred there. Mapping options not working:

--regextrans2 's,^drafts,Drafts,' \
--regextrans2 's,^sent,Sent,' \
--regextrans2 's,^spam,Junk,' \
--regextrans2 's,^trash,Trash,'     # тут не нужно ставить обратный слэш так как является последним аргументом

or

--regextrans2 's,^Черновики,Drafts,' \
--regextrans2 's,^Отправленные,Sent,' \
--regextrans2 's,^Спам,Junk,' \
--regextrans2 's,^Корзина,Trash,'

or

--regextrans2 's,^&BCcENQRABD0EPgQyBDgEOgQ4-,Drafts,' \
--regextrans2 's,^&BB4EQgQ,BEAEMAQyBDsENQQ9BD0ESwQ1-,Sent,' \
--regextrans2 's,^&BCEEPwQwBDw-,Junk,' \
--regextrans2 's,^&BBoEPgRABDcEOAQ9BDA-,Trash,'

None of the options work, it still creates folders.
2. Second BUT
I encountered this when I decided to move another box (all boxes are working and users worked with them). After comparing folders, reading headers, during the already direct transfer of letters, it hung on the very first letter from the INBOX folder, it successfully transferred from the rest, it puzzled me!?!?!? I created two test mailboxes, on one from which the mail will be transferred, created several test letters in each folder (Inbox, Sent, Spam, Deleted, Drafts) Started the transfer and again "hung", it's interesting that when transferring without manually matching folders , the INBOX folder is transferred at the end of the process. He began to study, as it turned out, he successfully read all the headers and stopped to transfer exactly the body of the letter that was not read
Question - Which folder is missing?:

Host1 folders list:
[&BB4EQgQ,BEAEMAQyBDsENQQ9BD0ESwQ1-] = [Отправленные]
[&BBoEPgRABDcEOAQ9BDA-] = [Корзина]
[&BCEEPwQwBDw-] = [Спам]
[&BCcENQRABD0EPgQyBDgEOgQ4-] = [Черновики]
[INBOX]

Host2 folders list:
[Drafts]
[INBOX]
[Junk]
[Sent]
[Trash]

Answer - The "Unread" folder is missing in case of host1 and "new" in case of host2 (new is not displayed in the roundcube web face, but is a folder in vmail)
Please help me to solve these two problems:
1. How to map folders correctly on two servers?
2. How to transfer unread emails? (option: making all letters read is not suitable, since some users have up to 1000 or more unread letters. Users want to leave it like that)
Thank you in advance for your help!!!
PS: I apologize for the "many bukawak")))

Answer 1

The solution of two "BUT" is found, thanks to dimonchik2013 for the hint on the first point:
did you try it on p1?
serverfault.com/questions/598364/imapsync-doesnt-s...
As a result, the working sync_loop_unix.sh script looks like this:

#!/bin/sh
echo Looping on account credentials found in file.txt
echo

{ while IFS=';' read  h1 u1 p1 h2 u2 p2 fake
    do
        { echo "$h1" | egrep "^#" ; } > /dev/null && continue # this skip commented lines in file.txt
        echo "==== Starting imapsync from host1 $h1 user1 $u1 to host2 $h2 user2 $u2 ===="
        ./imapsync --host1 "$h1" --user1 "$u1" --password1 "$p1" \
                    --host2 "$h2" --user2 "$u2" --password2 "$p2" \
                    --ssl1 \
                    --port1 993 \
                    --folderfirst INBOX \
                    --regextrans2 "s/&BB4EQgQ,BEAEMAQyBDsENQQ9BD0ESwQ1-/Sent/" \
                    --regextrans2 "s/&BBoEPgRABDcEOAQ9BDA-/Trash/" \
                    --regextrans2 "s/&BCEEPwQwBDw-/Junk/" \
                    --regextrans2 "s/&BCcENQRABD0EPgQyBDgEOgQ4-/Drafts/" \
                    --regexflag 's/\\Unseen//g' \
                    --useheader Message-Id
                 "[email protected]"
        echo "==== Ended imapsync from host1 $h1 user1 $u1 to host2 $h2 user2 $u2 ===="
        echo
    done
} < file.txt

And now let's sort it all out:
1. The file.txt contains data about the mail accounts that you will synchronize, each line is two mailboxes synchronized with each other:

host001_1;user001_1;password001_1;host001_2;user001_2;password001_2;
host002_1;user002_1;password002_1;host002_2;user002_2;password002_2;
host003_1;user003_1;password003_1;host003_2;user003_2;password003_2;

Example:

# вариант с использованием посредника, то есть на другой машине выполняете скрипт
imap.mail.ru;[email protected];[email protected];imap.newdomain.ru;[email protected];[email protected];
imap.mail.ru;[email protected];[email protected];imap.newdomain.ru;[email protected];[email protected];
... и т.д.
# если выполняете скрипт локально на новом почтовом сервере
imap.mail.ru;[email protected];[email protected];localhost;[email protected];[email protected];
imap.mail.ru;[email protected];[email protected];localhost;[email protected];[email protected];
... и т.д.

In my case, I was executing the script directly locally on the new mail server.
2. In order for the folders Inbox, Sent, SPAM, Cart, Drafts to be correctly matched during synchronization and letters to get according to their folders, you need to install the additional module Unicode::IMAPUtf7 for Perl :
It is downloaded from CPAN installation is very simple, in the README described in great detail.
Then, to match folders, you need to write the following lines in the script:

...
# папка Отправленные
--regextrans2 "s/&BB4EQgQ,BEAEMAQyBDsENQQ9BD0ESwQ1-/Sent/" \
# папка Корзина 
--regextrans2 "s/&BBoEPgRABDcEOAQ9BDA-/Trash/" \
# папка СПАМ
--regextrans2 "s/&BCEEPwQwBDw-/Junk/" \
# папка Черновики
--regextrans2 "s/&BCcENQRABD0EPgQyBDgEOgQ4-/Drafts/" \
...

After that, these folders will not be duplicated for you.
3. In order for imapsync to correctly perceive unread messages in the Inbox, you need to add the line

...
--regexflag 's/\\Unseen//g' \
...

After all not "cunning" manipulations, letters will be transferred correctly.

Answer 2

Maybe it's easier to rename folders or move emails and delete unnecessary folders directly in the maildirs on the server, or look in your IMAP server for naming settings for special folders? If there is no such setting, then it is trivial to move. if the server does not give service folders with the same names, the folders will still be duplicated in the client programs and you will have to change them in the settings.