Two Way Folder Sync

[penguinpages]

I have a three node cluster file system. I am building out a lot of various things and so need simple method which will allow me to have open and change files from my Windows system to the servers. 99% of files are small txt files. Configuration files where I change with open files in notepad+ and save.. and then need to have it sync down to the three cluster nodes, so I can test

Ex: vi /sw2/clusterfiles/cluster_hosts -> Add some new DNS entry... and that common cluster DNS is called by all three nodes to shim into VMs as /etc/hosts when deployed.


So Source CAN be windows client writting to SMB share.
OR any of the three Centos Linux nodes via the common path

And based on time stamp ... latest will push and overwrite, push and delete , push and get new files (AKA two way sync with timestamp as key field)


This has no remote network needs.. all paths are mounted and so "local", I just need time based sync based on set interval, and or file write trigger.
Reason for sync is many of these files are for various projects, and I want to make sure they are all kept in sync automatically and if a cluster node dies they just catch back up... but also want a backup copy that is real time on that SMB external to cluster path. But to keep it simple... I just need two way sync based on time stamp change to local path that can run near real time (1min delay is ok as I know trigger based I have found no reasonable way to accomplish)

I have looked at ...
rsync (one way and does not look at remote to compare if need to pull, delete, update.
lsyncd https://groups.google.com/forum/#!topic ... GRjnRA9BQk
mirror (https://linoxide.com/linux-how-to/insta ... -way-sync/) but no option for CentOS and not sure it is active project
mirror (github project https://github.com/stephenh/mirror )

Open to suggestions. I will post what I find as it seems others have been trolling like myself for such as solution.

lsyncd.png (12.83 KiB) Viewed 4396 times

[tony_down_under]

Windows domain controllers have a shared folder called sysvol which does this replication, used for things like group policies to be sync between domain controllers.

Have you thought about not doing replication but instead, make a share available where all can connect to?

[penguinpages]

Yes.

The concern is that a lot of these files are very time sensitive on run. Cloud-init answer files. first-boot files... ansible scripts for k8 etc...

I tried to run them from my common switch share .. but its response had too much latency and I got weird timeout issues. So I brought back local... but issue is that nodes need to all have common files for commands against "cluster"

I tried rsync from one node to others but then issues arise with changes made on one node.. non-master... getting over written.

I guess I could come up with something REALLY ugly.. and do some kind of cron job every minute that does "ls -alh" greps for time stamp against some "last touch file with time stamp" and then does a copy and override, and delete if missing. AKA sync based on time stamp.

I just was hoping for something more "baked".

[hunter86_bg]

Why don't you setup a SAMBA which can be mounted by all 3 nodes and the Windows machine without any issues?

As far as I know Windows supports also NFS, so that is also an option.

[penguinpages]

response had too much latency and I got weird timeout issues. So I brought back local.

plus the aspect of wish to have data distributed and i have glusterfs, but it does not understand back end and front end... so remote clients and vms cant mount it.

[tony_down_under]

I Think you're sort of looking for something similar to dropbox/google drive sync across systems. I checked the owncloud docs yesterday but I was not sure about it. I think you would need an owncloud server storage and each of your systems would be clients in that respect.

There is a CLI client but I have not used it at all. This will copy the data to each of the machines in a "pull" https://doc.owncloud.org/desktop/2.1/advancedusage.html

I'm keen to know if you try it, how you get on. I do something similar in that I keep a set of text files as mini "how-to" guides for myself, organised in a directory tree; on google drive. On my computers I use the google drive client to make those how-to's available to me. I add and update them from any machine as local files. A edit/save will send the file to the other machines. And because they are local I can easily do a find by using grep to search on the text.

BTW I just mention owncloud server as I use it for making AD shares available via HTTPS.

[nardusg]

Maybe:

Osync:
A robust two way (bidirectional) file sync script based on rsync with fault tolerance, POSIX ACL support, time control and near realtime sync

https://github.com/deajan/osync

Ciao

Nar