Sync question about File List and Methodology

[rroot]

Hello,
I have a few quick questions (I hope) about the file list to be sync'ed and the method of sync'ing individual files of different size.

This question is concerning a folder which is being set up to only "Send".
1. How often is the File List updated with new (file or folder) changes?
1a. Is this a setting which is used within the "Transfer Mode". where you tell it the settings for either a "Regular sync" or the "Daily Sync"?

My Misunderstanding (maybe) & thoughts <bg>:
I set up a folder to be sync'ed daily at 01:00 (master). I manually sent the invite file to the computer which will Pull the folder (slave). I went to slave, opened PowerFolder and imported the invite file. Assigned a directory and let it run. The slave received the initial file list from master and that was it.
All well and good (I hope) as I actually wished the actual transfer of file data to happen at 01:00 hours.
From slave, I manually had power folder do a sync and pull all the files. All fine.
I gave the master about 30 min to add more data and change a few files contents within the folder being synced. From the slave, I checked the file sync and there were no files to be sync'ed.
But physically comparing the 2 folders, I can tell there is a diff in the length of a few files and the addition of a few more files on the Master.
PowerFolder is not reporting the new files and also not the individual file changes.
It always says, "No files available yet.... Fetching File List"

Is that by design and I an thinking incorrectly?
When I think of the setting within the setup, which I selected, Daily Synchronization with a date time of daily @ 01:00; I think that is the time when the files will be transferred? Not the time for which the File List gets updated with new information.
Am I incorrect in my thinking?

= = =

New Question, still keep in mind master / slave.
Concerning individual files which have changed within a sync'ed folder (master), in relationship to a duplicate file which has previously been synced on slave.
2. How does PowerFolder determine an individual file change on the master?
2a. Does PowerFolder send the entire file again or does it do an append to the slave?
2b. Does PowerFolder do a binary file comparison or just a crc between the individual files being sync'ed between master / slave?

Thanks

[B.A.]

Currently we are doing a lot of changes on the file detection(will be instant by default soon but still scalable) , therefore I will tell you how is the status of this as good as i know and a developer will recheck what I wrote but that takes some time.

1.)
At 3.1.5 the local file lists are updated in the interval set in the transfer mode.
Transfer modes always only are local settings. They determine if the computer is scanning for new files on its hard drive or downloading files please see:
http://wiki.powerfolder.com/wiki/Transfer_Modes

1.b)
yes in the custom settings panel of the transfer modes (on the settings tab of a folder /press new) you can control the timed settings and interval but this still depends on the base mode you selected. E.g changing the sync interval by a profile which is not scanning for changes will not do something.

2.)
The master is doing a local scan (based on your settings, by deafult every 5 min on automatic sync... see wikilink above ). After this other computer will be able to download new files but dont need: see manual sync. But as mentioned this is changing.

2a.
PowerFolder is trying to update the file by only sending the changed part by default, you can change that under preferences/advanced delta sync so always the complete file is send. Some compressions/file formats will make this difficult so the complete file needs to be resend.

2b.
Not sure what is crc. As far as I know the file get split into very small peaces all of them have a checksum if one of the checksum is changing the part gets listed as changed.
Also I am not sure if we have something as master/slave. Since we do not have a herachy. Hierarchical right structures/management will be ready in the server version which is in beta(1) testing.
Sorry for this insufficient answer but I am not a developer. A dev will comment this soon.

I hope I did not got you wrong with the other answers.

[rroot]

Bernhard,

Thank You so much for the reply.
Your explanation resolved a few things in my head about the Flow of the program.
Explains a lot to me and makes sense in what I am seeing happen.
It also explains why I am seeing a lot of traffic between my 2 computers.
Remember I am just testing this out on my home lan between a few computers. Nothing on the internet as of yet. I wish to understand everything I can before I institute it.

I was using the wording Master/Slave to represent the 2 computers.
The Master is the computer on which I initially made a Folder to be Synced with another Computer, it would be a Send Only.
The Slave is the computer on which I wished to receive the files from the Master. It would be a Receive Only, with file deletion properties set. If I am thinking correctly, the Slave, receive only can tell if a file has been deleted, and will delete the file automatically when the file.list sync says it is not on the Master anymore.

Is that clear? It is another late night for me, 04:30 or so in the AM.

I must have a setting in my last test set incorrectly.
Your explanation of how it works, lets me know a few settings to set. But I am a little unsure about another setting.

Again I use the Master / Slave. What is the wording you use, and I will use it from now on.
I set the folder to be synced at 01:00, exact same scenario as in the first question I wrote. Same setup.
Remember I manually pulled all files Before the sync at 01:00.
I have a file which was deleted on the master sometime in between when I first pulled and when PowerFolder did a sync at 01:00.
This file was deleted by the program which is controlling the folder.
The Slave computer had a file list which contained the file.
The Master does not have the file anymore.
The Slave is continually attempting to pull the file.

I must have a setting set incorrectly, or just did not set it as it should be.
I would think PowerFolder SERVICE (on Master) would see the file is no longer there and would tell the Slave (who is trying to pull the non-existent file) to delete the file request.

I am sure it is just a setting I have set incorrectly.

Honestly, I am reading as much as I can. But some of the wording is throwing me off a little bit and some things I am not catching. Although I am sure I have read over them. Or maybe the inner workings are not as self evident to a new user as they may be to an experienced user.

Take the explanation you gave me about 2a. "under preferences/advanced delta sync", I had no idea what the wording "delta sync" was. I know I have read about it, but did not know what it actually did, or it's use within PowerFolder's context, now I know and understand.

Part of the way to learn the program is to jump in and run it through it's paces.
See what happens and learn what all the switches and settings do.

Also the CRC is a Cyclic Redundancy Check, it is a checksum method as you described PowerFolder doing.
http://en.wikipedia.org/wiki/Cyclic_redundancy_check
I just will have a lot of huge data files which I don't wish to be transferred over and over again.
As you described a checksum and re-transmittal between the 2 folders is exactly what I need, and is very efficient.


And no you did not get me wrong with any answers. You explained a lot which I totally understood.
Thank You.

So in a nutshell, how I set the folder up initially to sync at 01:00. When the folder was created, PowerFolder made an initial file list, and will only update that file list at 01:00.
That is exactly what I was seeing from my aspect.
I would expect If I made a folder to sync and it was empty. The initial file list would have 0 files. If I was to put files in the folder, they would not be added to the file list until 01:00.
Then they would become visible to the computer wishing to Pull them
Understand & Thank You again.

Now to the singular file which has been deleted off my system.
The computer which is Pulling the files is still trying to Pull a non-existent file.
If Master (which initially had the file) is only set to Sync at 01:00
Then will the Slave be told that file is no longer there and stop asking for it. lol

OR do I have to change a setting within the Slave to recognize that a file is no longer there
And stop asking for it.

As it is now, this is a snip of the tcp/ip traffic.
I cut and pasted just small portions of the 2 packets.
- -
.powerfolder.message.RequestDownload..
Then is the same packet it tells the name of the file
D22D7FEC66A82F3296E5C02FFF506CB942928248.txj

The Master is replying
.powerfolder.message.AbortUpload...
Then is the same packet it tells the name of the file to abort
D22D7FEC66A82F3296E5C02FFF506CB942928248.txj
- -

But the Slave still keeps on trying. lol
Been doing this since 01:00
The Master is running as a service.
I know I could Stop the service and open PowerFolder and do something.
Exactly what I am unsure, but I would mess around and see what are within the right clicks and stuff.

lol I must me getting tired now.

Am going to snooze now and catch a few winks.
I will drop you a line later on & Thanks again
Robert

[Hannibal]

Some background information about the data integrity checks (checksum building) was added to our wiki: http://wiki.powerfolder.com/wiki/Delta_sync

About the continuous download requests from Slave to Master while the file is deleted on Master but not yet recognized because it has not been: 01:00:

This is intended behavior. The slave checks every minute if it is possible to download new files. If this fails, it keeps trying/checking for updates.

[rroot]

Sprajc,

Thank you for the reply & I understand now about a few more things.
Just a FYI, something you may wish to check out, I noticed this behavior in the past couple day while messing around with various settings.

1. Computer 1, sent a 5 min sync, send only, file deletions invite to computer
Computer 2 accepted and set the folder location.
I went to computer 1 and manually created a text file entered some data and closed it.
Computer 2 picked up the new file and transferred.
I changed the data in the text file on computer 1.
Computer 2 picked up the new data & transferred.
Went to computer 1 and deleted the text file.
Computer 2 picked up the deletion and deleted the file.
Went to computer 2 and manually created a text file within the folder location.
Computer 1 did not pick it up. Good
Went to computer 2 and noticed within PowerFolder, the new text file was listed.
I went to explorer and manually deleted the file. Looked within PowerFolder and the file was listed as Deleted in bold red.

OK crux of issue.
I could not delete the Red File Deleted indicator from within PowerFolder.
It was Not normal like the Red File Deleted indicators, that were listed when I deleted the files on computer 1 and computer 2 had the files listed in Red Bold and file deleted. For them, I could open another part of PowerFolder and delete the files from PowerFolder's Recycle Bin Listings.

This singular file was Not listed within the RecycleBin PowerFolder. It was Bold Red from within the main Files Tab > Local Files, or Local and Incoming.
Just a FYI.

Another FYI.
2. Same setup as before. I was on a 2 min sync though.
At times within computer 1 synced folder, there are files which are written to and kept open for 3 or 4 minutes awaiting data. If no data is received, the file will gracefully close and another will open when new data arrives.
The individual file may stay open for hours or a few minutes, until it is finally closed.

Crux of issue.
Computer 2 picks up the newly created file and ques it for download. It continually attempts to pull the data. but as each 2 min sync goes by, the file size is increased and the last 2 min of data is not pulled by computer 2. Then the 4 min & file size increase, then last 6 min & file size increase, etc...
I observed this over about 15 minutes that the file contents were not being actively downloaded. I manually stopped data input to the file on computer 1, the file was closed by the program and the last 15 min of data was pulled by computer 2 on the next sync.

This is on no consequence to me at the moment. I am going to set up for a single daily sync when the data will not be written to.
But just FYI as it may impact others. I am sure it is a file locking/unlocking issue. If I was on a server build, I think I would not have noticed this.
XP Pro here.

Another Thought.
On the "Files Tab" Select "Local Files Only" then select an individual file, Then click the "Details" button.
Within the Details pane, there is a listing for "Modified On" which shows the file date.

Is there any way of incorporating a listing which reflects the Date/Time the file was Downloaded?

Reasoning: Within this particular folder I am Syncing, all files have the exact date/Time, No matter when they were created, written to, closed.

Also within the File Listings, at the Top of the Pane, there is a row, which has, Name, Size, Modified By, Date, Availability. It would be nice to have a listing in there also, something like Transferred. Then if it was like the other folders, I could click on it and get a quick sort by time of Transfer.

Also if the capability was there to generate Transfer Time Stamps by Year/Month/Day then there would be no issue with sorting & being visually correct. I have not checked to see if sorting PowerFolder's by month/day/year has output is like this
1/??/????
11/??/????
12/??/????
2/??/????
Some programs sort like that.
I have only been messing around with a folder which has the same date/time.

I will see for myself when I do one that has random date/times/


Thank You