Upgrade utilities (Update checker and snapshot downloader/application)

[iFoggz]

These options not available for TestNet but a TestNet snapshot option can be added in future with ease!

Update Checker:

• Check github for latest version tag
• If daemon it will display on screen the update information
• If Qt a Message Box will appear with the information about the release
• Check on start up for new version
• Check every 24 hours via CScheduler

Snapshot Downloader:
Daemon:

• Start gridcoin daemon with argument -snapshotdownload and the snapshot process will start and once complete the daemon will continue to start up without any user actions needed.
• Start gridcoin qt with argument -snapshotdownload and the snapshot process will start (with Qt Progress updates) and once complete the Qt will continue to start up without any user actions needed
• Request snapshot download via Qt Menu which when confirm the user wishes to do so it will shutdown the wallet and perform the snapshot process and then once complete the user just has to restart the wallet and continue syncing up.

Snapshot Process:
Stage 1) Download snapshot.zip from gridcoin.us server (with realtime updates via % complete and download speed)
Stage 2) Verify the SHA256SUM of snapshot.zip against the SHA256SUM stored on gridcoin.us server
Stage 3) Remove previous blockchain data from the wallet data directory.
Stage 4) Extract the snapshot.zip file to data directory.

It's rather straight forward and works as intended. I've added _() to even the Qt side because we wont have the translations loaded for Qt but the daemon side does not use them directly. I've also adopted the .lock file check and fsbridge support to this as well.

This is a big PR and look forward to review and requested changes. I'm still looking to refactor and refine some things. This also means no more manual applying of snapshot for users.

Some Pictures:

[jamescowens]

Travis is failing because we need to add libzip as a dependency in the .travis.yml file.

[nathanielcwm]

Could the snapshot download also state the amount downloaded / total amount to download?

It would be nice if the snapshot window could state the size of the snapshot.

[iFoggz]

Could the snapshot download also state the amount downloaded / total amount to download?

It would be nice if the snapshot window could state the size of the snapshot.

yes

[iFoggz]

i'll have a few more changes and a couple refactors up here in a bit

[cyrossignol]

Some initial comments:

As we discussed in Slack, I think we need to at least add a configuration directive that disables the version check. Ideally, it would be nice for configure to completely disable the version check and snapshot code at compile time (--disable-version-check, --disable-snapshot) since these features are completely optional. Some users may not want their wallets connecting to out-of-network sites, and hardened services like scrapers, explorers, exchanges, and pools may want to ensure that this code doesn't exist in the final binary as a security precaution.

Eventually, we'll need to update build-unix.md documentation for the libzip dependency. I think we should also update release-process.md to document the exact format for tags and release names that the version check code depends on.

That said, I think these are great features! These will relieve a some of the recurring pain that new users encounter as they learn to use the wallet. I'm going to build this PR to try out the code and do some more in-depth testing.

[iFoggz]

as for the comments about -disable-version-check, --disable-snapshot):

i'll be adding the --disable-version-check (would --disable-update-check be better?)

snapshot is optional yes. but its only activated when a user does so want it to be used. Its not automatic. should a disable be added for it?

[iFoggz]

oh for the --disable-version-check. etc u meant whether or not the application uses it. that will be something new to me.

[iFoggz]

@cyrossignol i've been considering what you said. basically u want people to be able to disable both option through ./copnfigure but i think the snapshot will only be used if the user chooses to. but for update checks we can add a config arugment for that0

[iFoggz]

also a optiioms menu checkbox for disabling the update checker which would store it in config

[cyrossignol]

i've been considering what you said. basically u want people to be able to disable both option through ./copnfigure but i think the snapshot will only be used if the user chooses to. but for update checks we can add a config arugment for that0

@iFoggz It's just a suggestion--we don't need to figure that out now 🙂

The reasoning behind this is that it allows a user to remove some code in the binary that could be exploited if there's a vulnerability somewhere else in the program (like --with-miniupnpc=no, for example). It's a precaution that might be valuable for certain deployments that I described above.

Most users will want to use these features, so it should be enough to provide a config switch (or a checkbox in the GUI like you suggested).

[nathanielcwm]

also users may prefer using a different solution to get updates like apt, chocolatey, pacman etc.

…

On Tue, 29 Oct 2019 at 8:09 am, Cy Rossignol ***@***.***> wrote: i've been considering what you said. basically u want people to be able to disable both option through ./copnfigure but i think the snapshot will only be used if the user chooses to. but for update checks we can add a config arugment for that0 @iFoggz <https://github.com/iFoggz> It's just a suggestion--we don't need to figure that out now 🙂 The reasoning behind this is that it allows a user to remove some code in the binary that could be exploited if there's a vulnerability somewhere else in the program. It's a precaution that might be valuable for certain deployments that I described above. Most users *will* want to use these features, so it should be enough to provide a config switch (or a checkbox in the GUI like you suggested). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1576?email_source=notifications&email_token=AFMG25WEO2LYM4VQENN5O4TQQ55KLA5CNFSM4JFLDKZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECOZTLI#issuecomment-547199405>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFMG25REQCQHRJXPQWNEBMLQQ55KLANCNFSM4JFLDKZQ> .

[iFoggz]

once the pr of james gets put in development i'll rebase and apply the requested changes from above then push the commits with the changes. i've also added a update interval option as well. will add the check box in options when i can. good night

[iFoggz]

U saying u coverted to boost filesystem path and it had problems when passed to fsopen? I just made those changes locally as cycy requested.

…

On Tue., Oct. 29, 2019, 12:41 p.m. James C. Owens, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/upgrade.cpp <#1576 (comment)> : > + else + { + ZipFile = zip_fopen_index(ZipArchive, i, 0); + + if (!ZipFile) + { + ExtractStatus.SnapshotExtractFailed = true; + + LogPrintf("Snapshot (ExtractSnapshot): Error opening file %s within snapshot.zip", ZipStat.name); + + return false; + } + + std::string ExtractFileString = ExtractPath.string() + "/" + ZipStat.name; + + FILE* ExtractFile = fsbridge::fopen(ExtractFileString.c_str(), "wb"); I compile for Windows and tested the snapshot download and replace functionality, and indeed saw some anomalies with the path set to C:\Users\JCO\AppData\Roaming\GridcoinResearch-šus. It downloaded successfully, but then gave an error message that it was unable to clean up the existing database successfully. The blk000*.dat files were gone but not the txleveldb or its contents. When I changed the path back to normal, it was successful. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1576?email_source=notifications&email_token=ADFNKQFC3ACBIH6CI65BDTLQRCGYNA5CNFSM4JFLDKZ2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCJT2ZPI#discussion_r340287120>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFNKQB4L7DSZLR7BZMHBF3QRCGYNANCNFSM4JFLDKZQ> .

[jamescowens]

No I meant I changed the actual path to the data on the machine to a standard path. This confirms the issues were a unicode conversion issue.

[iFoggz]

this will be fixed up tomorrow. got halloweening to do with kids but its almsot complete on my side

[iFoggz]

yes i will

[iFoggz]

For QMessageBox you cannot use scrollbars. but however after 4.2 they added DetailedText. which we can have the user click show details to see the change log :) i just push that but i plan to refactor it a little but and include version in main part. give me a little bit to do that just getting lil one to bed

[cyrossignol]

I really like the details dialog! Feels a little bit too small right now, though.

[cyrossignol]

Occasionally, the GUI snapshot process fails on Windows with an error similar to this when removing the old blockchain files:

Snapshot (CleanupBlockchainData): Exception occured: boost::filesystem::remove: The process cannot access the file because it is being used by another process: "C:\Users\Cy\AppData\Roaming\GridcoinResearch\txleveldb\000018.ldb"

I haven't had time to look closely into it yet, but it seems as if LevelDB is not releasing a file handle when the main application terminates and the snapshot procedure begins. It appears to happen in about one out of four trials. I'm not sure how to solve this yet...just wanted to bring it up so we keep it in mind.

Edit: this appears to be the same issue that @jamescowens encountered earlier. This does not look like a symptom of a path encoding problem—as seen above, I observed the error with the default data path that contains no special characters.

[iFoggz]

i had to add click the show details to widen the window. there is a setMinimumSize function but i'm not exactly too sure how it works yet or if it will work.

[iFoggz]

and LevelDb issue thx for bring that up. it would be nice if we eventually moved all the database stuff up to date to bitcoin so in theory we can release all in a reset and then wallet can just reload itself after snapshot applied. bitcoin has pointers there and a reset function which makes new i believe if i remember correctly. since they allow even wallet loads/unloads etc. it may be able to work with blockchain files as well. its a lil out of my area maybe you can look at that

[jamescowens]

I bet I know what is happening. The scraper thread is created by ThreadScraper in net.cpp. This thread is independent of the GUI. If QApp->Quit() is called, this will kill all of the UI threads, but if the scraper thread is busy doing something it will not collapse right away. For normal nodes this should not be an issue I think as the scraper really doesn't do long term activity, but for a node acting as a scraper also (like ours on testnet), if the scraper is downloading and processing files, it could be a number of seconds until the scraper thread ends. You can see this by watching the scraper log continue through the file download if it is busy downloading a file when the GUI exit (close window) is called. Because leveldb is really scoped at the main level, it will remain resident until the scraper thread collapses and the application ends.

[cyrossignol]

A few more rough edges to think about:

[cyrossignol]

We're not deleting the snapshot.zip file at all when the user starts the snapshot process from the GUI menu item.

[iFoggz]

willl look at this

[iFoggz]

this the one u are referring to

[iFoggz]

will address new comments shortly making a early dinner

[iFoggz]

I bet I know what is happening. The scraper thread is created by ThreadScraper in net.cpp. This thread is independent of the GUI. If QApp->Quit() is called, this will kill all of the UI threads, but if the scraper thread is busy doing something it will not collapse right away. For normal nodes this should not be an issue I think as the scraper really doesn't do long term activity, but for a node acting as a scraper also (like ours on testnet), if the scraper is downloading and processing files, it could be a number of seconds until the scraper thread ends. You can see this by watching the scraper log continue through the file download if it is busy downloading a file when the GUI exit (close window) is called. Because leveldb is really scoped at the main level, it will remain resident until the scraper thread collapses and the application ends.

interesting, surprised the scraper thread is not apart of the same thread group. when the client exits it waits for all threads to join after interrupt. then once that is complete it'll do the snapshot part.

[jamescowens]

@iFoggz It is part of the same thread group. (It is kicked off from net.cpp).

[cyrossignol]

I don't think it was a thread problem--an old change commented-out the line that closes LevelDB on shutdown, so it left the file handles open. I will finish testing this PR tonight!

…

On 11/15/19 12:44 PM, James C. Owens wrote: @iFoggz <https://github.com/iFoggz> It is part of the same thread group. (It is kicked off from net.cpp). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1576?email_source=notifications&email_token=ABAVQECGXTFJQ7RT72JRQETQT3UXLA5CNFSM4JFLDKZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEGLJEA#issuecomment-554480784>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAVQEAWJS2IUTJ3LYH4GILQT3UXLANCNFSM4JFLDKZQ>.

[cyrossignol]

Looks like everything has been resolved. Just one issue on Windows...

[cyrossignol]

I think this is good to go!

[jamescowens]

Merging.