Inotify for windows #1451
Labels
Comments
|
Owner changed to @alexbrainman. Status changed to Accepted. |
I have never used anything like that myself, and I don't know how it could work. But quick search suggests ReadDirectoryChangesW could be a good fit here. This is very low priority for me, but if you're interested to try and implement it yourself http://golang.org/doc/contribute.html, everyone will help you along. Alex Labels changed: added os-windows. Status changed to HelpWanted. |
My attempt at this can be found at http://golang.org/cl/4188047/. If anybody can review and/or test it that would be great! |
I've started on an API that should work with the subset of functionality provided by both windows and linux, callback-based rather than channel-based. Currently only supports linux, but making watchers for other operating systems should be trivial; they need only satisfy the Watcher interface (see the README). Supports Modify, Delete, and Create events currently. A Move event is planned, but not implemented for InotifyWatcher. Let me know if you have any feedback or concerns about this API. Repository is at https://github.com/crazy2be/fsmon. |
I don't use this functionality myself, so it is hard for me to comment. But you're welcome to start a discussion on golang-nuts, if you have some proposal in mind. FYI, there is this change http://golang.org/cl/4188047/ that has not come to anything yet. Alex |
I am keen to try the changes in http://golang.org/cl/4188047/ if I can get my Windows build working. What's the best way to move this toward release? |
> I am keen to try the changes in http://golang.org/cl/4188047/ if I can get my Windows build working. If you're asking how to build go for Windows, then you need mingw tools. You could download them yourself: http://code.google.com/p/go-wiki/wiki/WindowsBuild, or you could get pre-build zip file that Joe Poirier prepares https://groups.google.com/d/topic/golang-nuts/u9h4TSs5mRk/discussion. Then you need to fetch go repo with mercurial and build it. Alternatively, I use linux/386 to develop and just cross-compile to windows/386. > What's the best way to move this toward release? As far as I remember, CL didn't go anywhere, because we couldn't decide on what the interface should look like, so it is simple, reliable and works across of different OS and OS-apis. Russ suggested we discuss that on golang-nuts. And that is where it stopped. You could start discussion again if you have good suggestion on how to implement it. Alex |
The difficulty here is that the interfaces provided by different operating systems differ so much that making a single, coherent, interface is going to potentially result in loosing a significant amount of functionality. In particular, we loose the advantages each system provides, making implementing something that makes full use of the APIs (e.g. a search indexing engine) quite difficult. However, that being said, we can easily create an API that supports the most useful subset of functionality: watching arbitrary directories for create, delete, and modify events. The situation on Linux is probably the rosiest, which is part of the reason os/inotify already exists. The inotify api provides events of almost every variety, for any directory, in a reasonably coherent API. The go wrapper on this is pretty nice, abstracting the low-level API surprisingly nicely. On Windows, things are still pretty nice, if the function calls are a bit longer and uglier :P. ReadDirectoryChangesW() appears to support the subset of functionality I described above almost perfectly, and even has the additional feature of being able to watch sub-directories natively, no additional directory-walking code required. I wasn't able to figure out how you register to receive and then receive events, but It seems possible that ReadDirectoryChangesW() returns all changes since the folder was opened or it was last called. On Mac OSX, the situation is UGLY. The API is completely and totally different from the ones on Windows and Linux, and, it seems, designed for a completely different usage case. It was implemented in order to allow the search indexer to keep up-to-date on file changes, and was initially private to only that program. Since then, the FSEvents API has been added, which allows external applications access to the same information that the search indexer gets, filtered by top-level directory. This is recursive in nature, and it is impossible to make it non-recursive (although you can obviously ignore events from subfolders). Because you can ignore information, that is not a big issue. The bigger issue is that this API does not provide you with any information on *how* a file changed, telling you only that it changed. The jnotify library gets around this by checking for changes itself, but this has obvious disadvantages in that you can only see changes within the directory, and you're liable to miss events if there are multiple events within a short period. Moves seem especially difficult to guess, i'm not sure how jnotify gets around this. All in all, I think that we should provide the low-level (syscall) and medium-level (os/inotify) interfaces to each library, in case application developers want to take advantage of this additional functionality on a particular platform. The wrapper library should provide only basic functionality, like detecting modifications, moves, creations, and deletions. I'm undecided whether we should support recursive watching, it certainly has quite a few uses I can think of, and is supported directly on all OSes except Linux. The interface for this library is tricky to decide on. I started on mine (http://github.com/crazy2be/fsmon) with Callbacks, simply because that seemed like it would work best for the way I was using the library. However, channels are also an option, perhaps divided like this: watcher := fsmon.NewWatcher(true) watcher.AddWatch("foo") str, str := <-watcher.Moves err := <-watcher.Errors str := <-watcher.Deletes str := <-watcher.Creations str := <-watcher.Modifications Let me know of your ideas and thoughts on this, @michael. |
This is all very interesting, but there are 7 people (who monitor this issue) that can see your post. That's not enough. Please, post your suggestion on dolang-dev or golang-nuts. You should receive better response there. I know nearly nothing about the technology. Might be able to help with implementation. But we need to decide on api first. Thank you. Alex |
Thanks for all the feedback. We do need to move this to golang-nuts. Alex, thanks for the links - I do have all that installed, just having some errors in some tests. Hopefully will figure it out soon. crazy2be, to receive events that a directory or subdirectory was modified in Windows you use FindFirstChangeNotification/FindNextChangeNotification. It returns a handle that you wait on using WaitForSingleObject or WaitForMultipleObjects. You still have to use ReadDirectoryChanges to get the specific changes - these other functions just give your thread a way to wake up when changes happen. I personally like the initial inotify interface using one channel. Using a channel seems more idiomatic for Go. Because you can filter the events, it seems odd to break them into multiple channels - you'd have channels that never have messages. Plus, multiple events could happen on the same file, and I am not 100% sure but I think the events can be combined. I'll start a topic on golang-nuts. |
|
Comment 16 by [email protected]: I don't think ReadDirectoryChangesW is correct API to be using for this sort of general facility. NTFS Change Journals are the preferred method and are used by the OS for other services (eg. indexing). |
|
Comment 17 by [email protected]: To amend what scot said above - ReadDirectoryChangesW is broken in that it misses changes on large change volumes. NTFS change journals are the only reliable method on Windows. |
Interestingly enough, change journals (see http://msdn.microsoft.com/en-us/library/aa363798(v=vs.85).aspx) are also quite alike what is offered on MacOSX. Not sure if a similar alternative exists for Linux. Ultimately, before deciding what the API will look like, we need to decide what capabilities will be most important for people using the API, and what some of the common usage scenarios are. In light of this, I think it might help if interested parties make a brief statement of their intended use of the library, and what features it would enable in their application. What do you want to use such a file change notification library for? |
Link got mangled... Go here: http://msdn.microsoft.com/en-us/library/aa363798(v=vs.85).aspx |
Just a note. There is some new activity on http://golang.org/cl/4188047/ and http://golang.org/cl/5248062/. Please, comment if you are interested. Alex |
This issue was closed by https://code.google.com/p/go/source/detail?r=04345e5969cf00ae17e0fa3d35a4116d086286a9. Status changed to Fixed. |
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
by xavier.mehaut:
The text was updated successfully, but these errors were encountered: