View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000218 | VisualDiffer | General | public | 2013-12-30 21:14 | 2014-02-23 09:27 |
| Reporter | Mr.Kiwi | Assigned To | admin | ||
| Priority | high | Severity | major | Reproducibility | have not tried |
| Status | closed | Resolution | fixed | ||
| Platform | Mac | OS | Mavericks | OS Version | 10.9.1 |
| Product Version | 1.5.7 | ||||
| Fixed in Version | 1.5.9 | ||||
| Summary | 0000218: Copy and Sync folders set invalid file time on samba volumes | ||||
| Description | Action "copy to right" and "sync to right" not respect the origin file timestamp and not set the timestamp well. | ||||
| Steps To Reproduce | Assumption: - On left side a folder with some files exists - On right side this folder and the content is complete missing than - select folder on left side an use action "copy to left" or "sync to right" - as result all files and folders will be copy, but all file time stamps are set invalid (current time) - As result, all files/folders will be shown as different | ||||
| Tags | No tags attached. | ||||
|
|
Which type of filesystems (HFS, NTFS, FAT32) are on left *AND* right? Are disk removable? On my mac formatted with (HFS case insensitive) everything works fine Please give me more details because I can't reproduce this bug |
|
|
The filesystem is an local mounted NAS, smbfs. With early version of VisualDiff (and MacOS 10.7), "Sync" has set timestamp well, but "Copy" not. |
|
|
Ok, I never tested with smbfs this should be the problem. "Sync" internally calls "copy" so it is very strange these actions works differently, I will try to reproduce the bug thanks |
|
|
OK, I can confirm randomly the date/time isn't set. The bug isn't strictly related to VisualDiffer but how Apple system calls work, I'm searching a workaround. Please apologize for this inconvenient |
|
|
First) Happy New Year and best greetings from Germany! Second) I test follow in a terminal: my-mac: kiwi$ mount ... //GUEST:@my-nas/Public on /Volumes/Public (smbfs, nodev, nosuid, noowners, mounted by kiwi) my-mac: kiwi$ cd /Volumes/Public/Sample my-mac: kiwi$ touch readme.txt my-mac: kiwi$ ls -l -rwxrwxrwx 1 kiwi staff 0 2 Jan 2014 readme.txt my-mac: kiwi$ touch -t 1206232233.44 readme.txt my-mac: kiwi$ ls -l -rwxrwxrwx 1 kiwi staff 0 23 Jun 2012 readme.txt So, you can see, that the unix command "touch" can set the timestamp well. May be, you should use "uptime()" to set the timestamp. |
|
|
Sorry, my mistake: I mean "utime()" not "uptime()" |
|
|
Hi, happy new year also to you :) The bug is strange, if I use the VisualDiffer touch everything works fine (it does exactly the same code to change attributes). The creation time is set correctly but the modification time is "reset" to the copy time but both dates are set in same call, really weird |
|
|
Your hint is correct, using utimes (utime is deprecated) seems to work. I suspect the cocoa API have some bug I would ask if you can help me to test this fix, I would send to you an alpha version so you can test yourself. |
|
|
Yes, sure, I can test it. |
|
|
At the link below you can find the version with the fix. ** Please consider this is an alpha ** 1. Do not overwrite the Mac App Store version 2. Unzip where you prefer, also your home directory 3. run as usual 4. test it but don't do with important or not backed up data (I repeat this is an alpha) Thanks again to help me on 2nd January :) http://cl.ly/220f2f302H18 |
|
|
I have execute a test (as std-user, without admin privilege), results: my-mac: kiwi$ mount … //GUEST:@my-nas/sandbox on /Volumes/sandbox (smbfs, nodev, nosuid, noowners, mounted by testuser) Next, I create on NAS two „folder-1“ and „folder-2“. Next, I copy from Apple-Default-Document „About Downloads.pdf“-Bundle to „folder-1“ Now: - I start VisialDiffer 1.5.7 (AppStore version) - Than I check behavior of „copy to right“ and „sync“ - and yes, bug is reproducable Next: - I start new VisualDiffer -bugfix218 Version (result --> app start well) - I check with „lsof“ - yes the application runs from correct path ($HOME) - Now I check output of „About VisualDiffer“ dialog - report version 1.5.7 - Now I choose „compare file timestamps and size“ - Now I select on left side the folder and choose „copy to right“ action, as result I get - some files have new timestamps (About Downloads.lpdf/Contents/Resources/ca.lproj/InfoPlist.strings) - some files have an timestamp (+ 1 second) and - some files have the correct (old) timestamp (from left side) - Now I double-check the result, I choose „Compare file content only“ - as result, no difference are shown, I assume, copy operation works well, only timestamp is invalid - Now I select „compare file timestamps and size“ again and clean content of „folder-2“ - Than select content of „folder-1“ and execute „sync to right“ action - same result like „copy to right“, some timestamps are incorrect. Finally with Finder: I choose an original file (left side) and show information with Finder about this file, and I see that creation,access,modification timestamp are equal. I choose an „sync“ file (right side) and show information with Finder about this file, and I see that „creation“ time is correct (like left side), but „modification“ and last-access time are newer (invalid). Additional I see: my-mac: kiwi$ cd /Volumes/sandbox my-mac: kiwi$ ls -la folder-1/About\ Downloads.lpdf/Contents/Resources/hr.lproj/ -rwxrwxrwx 1 testuser staff 299334 2 Jan 13:04 About Downloads.pdf -rwxrwxrwx 1 testuser staff 111 2 Jan 13:04 InfoPlist.strings ls -la folder-2/About\ Downloads.lpdf/Contents/Resources/hr.lproj/ -rwxrwxrwx 1 testuser staff 4096 2 Jan 13:47 ._About Downloads.pdf -rwxrwxrwx 1 testuser staff 4096 2 Jan 13:47 ._InfoPlist.strings -rwxrwxrwx@ 1 testuser staff 299334 2 Jan 13:47 About Downloads.pdf -rwxrwxrwx@ 1 testuser staff 111 2 Jan 13:47 InfoPlist.strings Conclusion: Somewhere seams to attach some file and create "._<filename>". |
|
|
Great test suite. The version with fix is 1.5.7 I haven't changed the number because official updated from MAS could be no longer "visible" ".lpdf" are special files in Apple world, they are resource forks so I think copying them on Windows filesystem can produce strange behavior. I try to make deeper tests and I'll keep you informed. Thank you very much |
|
|
About one second difference here also from finder does the same. From Finder I copy a file from mac mini to samba mapped disk and the copied modification time file is less than one second You can check using stat <filename>> In my case stat /Volumes/smbtest/sample01.txt 771751939 9849149156497250956 -rwxrwxrwx 1 dave staff 0 3 "Jan 2 00:00:00 2014" "Jan 1 07:11:32 2014" "Jan 1 07:11:32 2014" "Nov 9 13:18:44 2013" 16384 32 0 /Volumes/smbtest/sample01.txt and on mac mini stat /Users/dave/test_suite/smbtest/sample01.txt 16777217 6771460 -rw-r--r-- 1 dave staff 0 3 "Jan 2 14:59:02 2014" "Jan 1 07:11:33 2014" "Jan 2 07:11:41 2014" "Nov 9 13:18:45 2013" 4096 8 0 /Users/dave/trash/test_suite/smbtest/sample01.txt May you confirm??? This seems a problem related to OSX not VisualDiffer |
|
|
Oh, difficult: three points I found out: ------ first point -------------- Assume, /Volume/sandbox is a NAS (like Buffalo, Qsnap ...), mounted as SMB. Try follow shell commands: cd /Volumes/sandbox rm -rf /Volumes/sandbox/folder-1 rm -rf /Volumes/sandbox/folder-2 mkdir -p /Volumes/sandbox/folder-2 mkdir -p /Volumes/sandbox/folder-1/subfolder-1 mkdir -p /Volumes/sandbox/folder-1/subfolder-2 touch /Volumes/sandbox/folder-1/subfolder-1/content.txt touch /Volumes/sandbox/folder-1/subfolder-2/content.txt echo "hello" > /Volumes/sandbox/folder-1/readme.txt sleep 5 ( cd /Volumes/sandbox/folder-1 && cp -Rp subfolder-1 subfolder-2 readme.txt ../folder-2 ) As result, VisualDiffer shown no diffs, all timestamps are preserved well by "cp -Rp ..." ------------- second -------------- Use follow command lines cd /Volumes/sandbox rm -rf /Volumes/sandbox/folder-1 rm -rf /Volumes/sandbox/folder-2 mkdir -p /Volumes/sandbox/folder-2 mkdir -p /Volumes/sandbox/folder-1/subfolder-1 mkdir -p /Volumes/sandbox/folder-1/subfolder-2 touch /Volumes/sandbox/folder-1/subfolder-1/content.txt touch /Volumes/sandbox/folder-1/subfolder-2/content.txt echo "hello" > /Volumes/sandbox/folder-1/readme.txt sleep 5 ( cd /Volumes/sandbox/folder-1 && rsync -av --exclude '.DS_S*' --exclude '._*' Folder-1/ folder-2 ) As result, VisualDiffer shown no diffs, all timestamps are preserved well by "rsync ..." ------------ last ------------- cd /Volumes/sandbox rm -rf /Volumes/sandbox/folder-1 rm -rf /Volumes/sandbox/folder-2 mkdir -p /Volumes/sandbox/folder-2 mkdir -p /Volumes/sandbox/folder-1 cp -rf $(HOME)/About\ Download.lprf /Volumes/sandbox/folder-1 sleep 5 ( cd /Volumes/sandbox/folder-1 && cp -Rp About\ Download.lpdf ../folder-2 ) Result: All timestamps of directories are preserved, but timestamps of files inside the *.lpdf are different. This behavior I see also on MacOS 10.5. ----- conclusion ----- rsync works correct in all cases. cp -Rp works correct for directories and subfolders, but failed with the *.lpdf file too. VisualDiffer also failed, if I copy/sync "subfolder-1" from left to right. So I can't confirm, that it is a MacOS problem only. The man page for "cp" is scary about the handling of preserve directory/xattr etc. So I will file a bug to apple for the "cp -Rp *.lpdf" issue. But for the other files (I test also with a bunch of *.pdf files) VisualDiffer failed, but cp and rsync works well. |
|
|
I'm trying to understand where is the bug but it seems in somewhere apple code, I've written a little snippet (a separated project) and works badly. I suppose cp uses some hack |
|
|
Just some little modification, I'm sure Cocoa classes refer to a wrong date (maybe a cached value), now I've modified the code to use *only* unix low level calls to determine and set modification time. I know I'm stressing your patience but if you can test it again would be great. The link below to new alpha http://cl.ly/2u1M3W243U33 thanks in advance |
|
|
|
|
|
Sorry for the delay. I have test your version 218-a2.zip today. I use the same test like described before. But I'm sorry, your app failed and the bug are not fixed. I attach a screenshot (screenshot-218-a2.png). Like you can see, some files have a correct timestamp. Additional, I take a look, how your app works. May be, I found your problem. For a correct behavior, you must first create the empty directory structure and touch the timestamps. (Look to the rsync behavior.) Then you must copy the files and touch there timestamps. Please note, if you copy a new file into an directory, then the OS modify the timestamp of the directory too. So, you must modify the timestamp again. After all is finished, you directory (modify) timestamp should be the newest timestamp of all files, that are members of this directory (in relation to the member creation timestamp). But you work with a recursive pipe, that may be the problem. |
|
|
First of all thank you very much to spend your time to make alpha testing for me. Folders timestamps are set after all children but when the files are on smbfs (I suppose when they aren't on local disks) something doesn't work. I will dedicate to this bug more time in next days, thanks again for your great feedback |
|
|
Good news, using a complex structure with many files now all modification timestamps are copied correctly at least based on my tests. I've searched on the web for similar problems without success but finally found a string matching on delta walker changelog http://www.softpedia.com/progChangelog/Deltopia-DeltaWalker-Changelog-85293.html "An alternate routine now attempts setting the Creation and Modified Date file attributes in cases when the destination file system, e.g. FAT32 accessible via samba, doesn't offer reliable implementation of these API." So I realized the problem isn't in my code but somewhere in samba land (or Apple code??) Now I've removed all Cocoa NSFileManager calls (also 'reading' not only 'writing' calls) and used only utimes() and I hope you can test again The link to new alpha is http://cl.ly/1r1I0T0c0B3x thanks in advance |
|
|
Ok, I test today the version 218-a3. - Result: better, but not correct. Details: I have on left side files/folders, like on screenshot. On right side, the folder (folder-2) is empty. I start VisualDiffer from "terminal.app" like ./VisualDiffer.app/Contents/MacOS/VisualDiffer I open an other tab (Cmd-T) with terminal.app (later for rsync calls) I select all files in left side in VisualDiffer and choose "copy to right" action. (All files will copy, after process is finished, VisualDiffer shown now diffs). Immediately after VisualDiffer finished, in "terminal.app" I execute "rsync -avn folder-1/ folder-2" --> I see no diffs, all is ok, only .DS_Store are different. BUT, wait 10 Seconds, then the "terminal.app" tab shown an short "spinning wheel". After them, execute "rsync -avn folder-1/ folder-2" again, and "BANG" timestamps are changed. ----- Same behavior you can get, if you "copy to right" after this process is finished, press "refresh" on toolbar of VisualDiffer. - "Refresh" operation seems to flush data and modify timestamps. ----- May be, you have an background process, that flush some cached operation? May be, "refresh" operations use NSFileManager? ----- Additional, if I execute VisualDiffer on "terminal.app" now I get the log message: CoreAnimation: warning, deleted thread with uncommitted CATransaction; set CA_DEBUG_TRANSACTIONS=1 in environment to log backtraces. |
|
|
> I start VisualDiffer from "terminal.app" like ./VisualDiffer.app/Contents/MacOS/VisualDiffer Ok, I do the same steps, on left populated folder with subfolders (it contains the About Stacks.lpdf), on right empty folder > BUT, wait 10 Seconds, then the "terminal.app" tab shown an short "spinning wheel". > After them, execute "rsync -avn folder-1/ folder-2" again, and "BANG" timestamps are changed. I can't reproduce this, the copy now preserves all timestamps and rsync doesn't report differences > Same behavior you can get, if you "copy to right" after this process is finished, press "refresh" on toolbar of VisualDiffer. This did happen with alpha2, now is fixed > May be, you have an background process, that flush some cached operation? May be, "refresh" operations use NSFileManager? There is a bkg thread but nothing is cached, I've checked before sending to you the alpha3 > CoreAnimation: warning, deleted thread with uncommitted CATransaction; set CA_DEBUG_TRANSACTIONS=1 in environment to log backtraces. At this time you can ignore this warning |
|
|
VD 1.5.8 is out but without a fix for this bug, I've a new alpha using a different approach so if you can help me again, you find the alpha 4 below. On my test env, a winxp shared folder, it works. thanks in advance for your patience Please before use the alpha, update to v1.5.8 from mac app store http://cl.ly/35423x1p2h30 |
|
|
I have test your new version VisualDiffer-Bug218-a4.zip. I use the same folder structure like last test. Follow I see as result: - Copy result have same content (md5sum check) - ok - Copy/Sync result of files have same creation time and modification time like origin files - ok - "Refresh" button after the "copy/sync" operation works well and no diffs are shown - ok - Folders have different times stamps - ups - Visual Differ ignores different timestamps for folders. More details to folder timestamps: - Visual differ show the timestamps correct. - A new created folder (after copy/sync action) have the current timestamp. - If I copy same with "rsync" the result folder himself have a timestamp like the origin, so the NAS can handle it well. Conclusion: - For files himself, the bug seem now to be fixed - For folder timestamp, we can discuss, if this is a new bug issue. - Please note, that I have read not all SMB implementations can handle folder timestamps well (depend on file systems). In my case, the NAS use XFS, that handle folder timestamps well, like the rsync test shown. But If you use FAT32, then the behavior of VisualDiffer (to ignore folder timestamps) may be correct. |
|
|
Please apologize me for delay in reply. The new folder modification timestamp isn't set by design not only on smb volumes. The idea behind this decision was because folder timestamps are not used for comparison but now I realize this decision isn't correct so I'll update timestamp for new created folders, too. I'm happy finally VD works on smb, I will publish v1.5.9 with this bug fixed very soon Just if you want I can post here, when available, the version with the correct setting of new created folders timestamp. Thanks again for your incredible useful feedback |
|
|
The version at http://cl.ly/1k0a2d0n3V1x copies timestamps for newly created folders, just if you want to take a look at it. If everything works I will publishing it in few days |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2013-12-30 21:14 | Mr.Kiwi | New Issue | |
| 2013-12-30 21:14 | Mr.Kiwi | Status | new => assigned |
| 2013-12-30 21:14 | Mr.Kiwi | Assigned To | => admin |
| 2013-12-31 08:11 | admin | Note Added: 0000424 | |
| 2014-01-01 14:32 | Mr.Kiwi | Note Added: 0000425 | |
| 2014-01-01 18:23 | admin | Note Added: 0000426 | |
| 2014-01-02 07:55 | admin | Note Added: 0000427 | |
| 2014-01-02 10:56 | Mr.Kiwi | Note Added: 0000428 | |
| 2014-01-02 10:58 | Mr.Kiwi | Note Added: 0000429 | |
| 2014-01-02 10:59 | admin | Note Added: 0000430 | |
| 2014-01-02 11:23 | admin | Note Added: 0000431 | |
| 2014-01-02 11:23 | admin | Severity | minor => major |
| 2014-01-02 11:23 | admin | Status | assigned => confirmed |
| 2014-01-02 11:25 | Mr.Kiwi | Note Added: 0000432 | |
| 2014-01-02 11:36 | admin | Note Added: 0000433 | |
| 2014-01-02 14:35 | Mr.Kiwi | Note Added: 0000434 | |
| 2014-01-02 14:46 | admin | Note Added: 0000435 | |
| 2014-01-02 15:06 | admin | Note Added: 0000436 | |
| 2014-01-02 16:15 | Mr.Kiwi | Note Added: 0000437 | |
| 2014-01-02 16:21 | admin | Note Added: 0000438 | |
| 2014-01-02 16:22 | admin | Note Edited: 0000438 | |
| 2014-01-03 13:15 | admin | Note Added: 0000439 | |
| 2014-01-05 16:44 | Mr.Kiwi | File Added: screenshot-218-a2.png | |
| 2014-01-05 17:00 | Mr.Kiwi | Note Added: 0000440 | |
| 2014-01-05 17:17 | admin | Note Added: 0000441 | |
| 2014-01-06 13:55 | admin | Note Added: 0000446 | |
| 2014-01-06 15:07 | Mr.Kiwi | Note Added: 0000447 | |
| 2014-01-06 16:46 | admin | Note Added: 0000448 | |
| 2014-01-06 16:49 | admin | Note Edited: 0000448 | |
| 2014-01-19 10:06 | admin | Note Added: 0000467 | |
| 2014-01-23 18:15 | admin | Summary | Copy and Sync folders set invalid file time => Copy and Sync folders set invalid file time on samba volumes |
| 2014-01-26 19:26 | Mr.Kiwi | Note Added: 0000470 | |
| 2014-01-29 17:46 | admin | Note Added: 0000471 | |
| 2014-02-05 17:45 | admin | Note Added: 0000476 | |
| 2014-02-23 09:27 | admin | Status | confirmed => closed |
| 2014-02-23 09:27 | admin | Resolution | open => fixed |
| 2014-02-23 09:27 | admin | Fixed in Version | => 1.5.9 |
